Dynamically filling parameters from a file in a Jenkins pipeline - jenkins

TL;DR:
I would like to use ActiveChoice parameters in a Multibranch Pipeline where choices are defined in a YAML file in the same repository as the pipeline.
Context:
I have config.yaml with the following contents:
CLUSTER:
dev: 'Cluster1'
test: 'Cluster2'
production: 'Cluster3'
And my Jenkinsfile looks like:
pipeline {
agent {
dockerfile {
args '-u root'
}
}
stages {
stage('Parameters') {
steps {
script {
properties([
parameters([
[$class: 'ChoiceParameter',
choiceType: 'PT_SINGLE_SELECT',
description: 'Select the Environemnt from the Dropdown List',
filterLength: 1,
filterable: false,
name: 'Env',
script: [
$class: 'GroovyScript',
fallbackScript: [
classpath: [],
sandbox: true,
script:
"return['Could not get The environemnts']"
],
script: [
classpath: [],
sandbox: true,
script:
'''
// Here I would like to read the keys from config.yaml
return list
'''
]
]
]
])
])
}
}
}
stage("Loading pre-defined configs") {
steps{
script{
conf = readYaml file: "config.yaml";
}
}
}
stage("Gather Config Parameter") {
options {
timeout(time: 1, unit: 'HOURS')
}
input {
message "Please submit config parameter"
parameters {
choice(name: 'ENV', choices: ['dev', 'test', 'production'])
}
}
steps{
// Validation of input params goes here
script {
env.CLUSTER = conf.CLUSTER[ENV]
}
}
}
}
}
I added the last 2 stages just to show what I currently have working, but it's a bit ugly as a solution:
The job has to be built without parameters, so I don't have an easy track of the values I used for each job.
I can't just built it with parameters and just leave, I have to wait for the agent to start the job, reach the stage, and then it will finally ask for input.
Choices are hardcoded.
The issue I'm currently facing is that config.yaml doesn't exist in the 'Parameters' stage since (as I understand) the repository hasn't been cloned yet. I also tried using
def yamlFile = readTrusted("config.yaml")
within the groovy code but it didn't work either.
I think one solution could be to try to do a cURL to the file, but I would need Git credentials and I'm not sure that I'm going to have them at that stage.
Do you have any other ideas on how I could handle this situation?

Related

Jenkins Pipeline - Git checkout collision

We have a multi-stage pipeline in our CI, and some of the stages have their own nested stages that are parallelized and may run on the same or different agents (we request a certain agent label).
As with most CI pipelines, we build our artifacts and deploy and run our tests later.
As the pipeline may take some time to complete, we had an issue where new commits that are merged to our master branch may be picked up in the later stages, and it creates an incompatibility between the pre-packaged code and the new checkout-out one.
I'm currently using the skipDefaultCheckout directive and added my own function to checkout the commit SHA1 that is set in the GIT_COMMIT in every one of the parallel stages
void gitCheckoutByCommitHash(credentialsId, gitCommit=GIT_COMMIT) {
script {
println("Explicitly checking out git commit: ${gitCommit}")
}
checkout changelog: false, poll: false,
scm: [
$class: 'GitSCM',
branches: [[name: gitCommit]],
doGenerateSubmoduleConfigurations: false,
extensions: [
[
$class: 'CloneOption',
noTags: true,
shallow: true
],
[
$class: 'SubmoduleOption',
disableSubmodules: false,
parentCredentials: true,
recursiveSubmodules: true,
reference: '',
trackingSubmodules: false
],
],
submoduleCfg: [],
userRemoteConfigs: [[
credentialsId: credentialsId,
url: GIT_URL
]]
]
}
The problem I'm facing is that sometimes, two or more of the parallel stages are trying to run on the same agent and perform the checkout, and I get an error that a process has already retained .git/index.lock, and the stage that is locked out fails.
Is there any way to work around that?
This is a sample pipeline
pipeline {
agent {
label 'docker_v2'
}
options {
timestamps()
timeout(time: 1, unit: 'HOURS')
}
stages {
stage('Prepare test environment') {
options {
skipDefaultCheckout()
}
steps {
gitCheckoutByCommitHash('some-creds-id')
}
}
stage('Parallel stuff'){
parallel {
stage('Checkout 1') {
agent {
label 'docker_v2'
}
options {
skipDefaultCheckout()
}
steps {
gitCheckoutByCommitHash('some-creds-id')
}
}
stage('Checkout 2') {
agent {
label 'docker_v2'
}
options {
skipDefaultCheckout()
}
steps {
gitCheckoutByCommitHash('some-creds-id')
}
}
}
}
}
}
The best way to solve this issue would be to only perform the checkout once, and then use stash, with unstash in later stages:
pipeline {
agent {
label 'docker_v2'
}
options {
timestamps()
timeout(time: 1, unit: 'HOURS')
skipDefaultCheckout()
}
stages {
stage('Prepare test environment') {
steps {
// you can use this:
gitCheckoutByCommitHash('some-creds-id')
// or the usual "checkoutScm"
stash name: 'sources', includes: '**/*', allowEmpty: true , useDefaultExcludes: false
}
}
stage('Parallel stuff'){
parallel {
stage('Checkout 1') {
agent {
label 'docker_v2'
}
steps {
unstash 'sources'
}
}
stage('Checkout 2') {
agent {
label 'docker_v2'
}
steps {
unstash 'sources'
}
}
}
}
}
}
This would also speed up your pipeline.
To prevent collisions in the workspace, you can also use one of the following:
workspace directive so that different parallel stages would use different workspaces; or
Define your agents so that there is only one executor per agent; or
Both of the above.

Jenkins File - Execute shell command in active choice script

everyone, I'm using plugin "Active Choices" in my Jenkins, I try to do something condition, and the return based on environment host that will be executed in shell command in my Jenkins file. the command like "echo ${BUILD_ENVIRONMENT}, echo ${BUILD_VERSION}..etc".
Already doing this but the list is still empty. now, what do I have to do to make this work? thank you.
JenkinsFile:
properties([
parameters([
[$class: 'ChoiceParameter',
choiceType: 'PT_SINGLE_SELECT',
name: 'ENVIRONMENT',
description: 'Select Environment',
script: [
$class: 'GroovyScript',
script: [
classpath: [],
sandbox: true,
script: '''
def getEnvVar(String name) {
return sh(script: "echo \${$name}", returnStdout: true).trim()
}
def env = getEnvVar("ENVIRONMENT")
if (envs != "testing") {
return ["stage", "dev", "prod"]
}else{
return ["testing"]
}
''',
],
]
]
])
])
Output:
im fix it with another solution, but maybe isnt the best option.
1st create a method to load the env file (i set the values on it), then define to jenkins global variable.
getEnvHosts = { ->
node {
script {
// Get the environment hosts file
def props = readProperties file: '/var/lib/jenkins/env/dast-service-env'
//define to jenkins global variable
env.BUILD_ENVIRONMENT = props['BUILD_ENVIRONMENT']
}
}
}
and then i call the "BUILD_ENVIRONMENT" with ${env.BUILD_ENVIRONMENT} like this in active choice parameter:
getEnvHosts()
properties([
parameters([
[$class : 'ChoiceParameter',
choiceType : 'PT_SINGLE_SELECT',
name : 'BUILD_ENVIRONMENT',
description: "Select Environment default is ${env.BUILD_ENVIRONMENT}",
script : [
$class: 'GroovyScript',
script: [
classpath: [],
sandbox : true,
script : """
def envs = "${env.BUILD_ENVIRONMENT}"
if (envs == "stage") {
choices = ["stage", "dev", "prod", "testing"]
}else if (envs == "dev") {
choices = ["dev", "prod", "stage", "testing"]
}else if (envs == "prod") {
choices = ["prod", "dev", "stage", "testing"]
}else{
choices = ["testing", "stage", "dev", "prod"]
}
return choices
""",
],
]
],
])
])

how to fail the jenkins build if any test cases are failed using findText plugin

I have a stage in Jenkins as follows, How do I mark the build to fail or unstable if there is a test case failure? I generated the script pipeline for textfinder plugin but it is not working. "findText alsoCheckConsoleOutput: true, regexp: 'There are test failures.', unstableIfFound: true" not sure where to place the textFinder regex.
pipeline {
agent none
tools {
maven 'maven_3_6_0'
}
options {
timestamps ()
buildDiscarder(logRotator(numToKeepStr:'5'))
}
environment {
JAVA_HOME = "/Users/jenkins/jdk-11.0.2.jdk/Contents/Home/"
imageTag = ""
}
parameters {
choice(name: 'buildEnv', choices: ['dev', 'test', 'preprod', 'production', 'prodg'], description: 'Environment for Image build')
choice(name: 'ENVIRONMENT', choices: ['dev', 'test', 'preprod', 'production', 'prodg'], description: 'Environment for Deploy')
}
stages {
stage("Tests") {
agent { label "xxxx_Slave"}
steps {
checkout([$class: 'GitSCM', branches: [[name: 'yyyyyyyyyyz']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'zzzzzzzzzzz', url: 'abcdefgh.git']]])
sh'''
cd dashboard
mvn -f pom.xml surefire-report:report -X -Dsurefire.suiteXmlFiles=src/test/resources/smoke_test.xml site -DgenerateReports=false
'''
}
}
}
}
All I did to make this request possible is as below:
added a post block of code below the steps block code.
post {
Success {
findText alsoCheckConsoleOutput: true, refexp: 'There are test failures.', unstableIfFound: true
}
}

Use Jenkins WORKSPACE environment variable in Jenkins declarative pipeline parameter

Is there a way to use Jenkins WORKSPACE environment variable in Jenkins declarative pipeline parameters?
Below attempt failed.
pipeline {
parameters {
extendedChoice description: 'Template in project',
multiSelectDelimiter: ',', name: 'TEMPLATE',
propertyFile: env.WORKSPACE + '/templates.properties',
quoteValue: false, saveJSONParameterToFile: false, type: 'PT_MULTI_LEVEL_SINGLE_SELECT',
value: 'Project,Template', visibleItemCount: 6
...
}
stages {
...
}
propertyFile: '${WORKSPACE}/templates.properties' didn't work either.
The environment variable can be accessed in various place in Jenkinsfile like:
def workspace
node {
workspace = env.WORKSPACE
}
pipeline {
agent any;
parameters {
string(name: 'JENKINS_WORKSPACE', defaultValue: workspace, description: 'Jenkins WORKSPACE')
}
stages {
stage('access env variable') {
steps {
// in groovy
echo "${env.WORKSPACE}"
//in shell
sh 'echo $WORKSPACE'
// in groovy script
script {
print env.WORKSPACE
}
}
}
}
}
The only way that worked is putting absolute path to Jenkins master workspace where properties file is located.
pipeline {
parameters {
extendedChoice description: 'Template in project',
multiSelectDelimiter: ',', name: 'TEMPLATE',
propertyFile: 'absolute_path_to_master_workspace/templates.properties',
quoteValue: false, saveJSONParameterToFile: false, type: 'PT_MULTI_LEVEL_SINGLE_SELECT',
value: 'Project,Template', visibleItemCount: 6
...
}
stages {
...
}
It seems that environment variables are not available during pipeline parameters definition before the pipeline actually triggered.

Active choice parameter with declarative Jenkins pipeline

I'm trying to use active choice parameter with declarative Jenkins Pipeline script.
This is my simple script:
environments = 'lab\nstage\npro'
properties([
parameters([
[$class: 'ChoiceParameter',
choiceType: 'PT_SINGLE_SELECT',
description: 'Select a choice',
filterLength: 1,
filterable: true,
name: 'choice1',
randomName: 'choice-parameter-7601235200970',
script: [$class: 'GroovyScript',
fallbackScript: [classpath: [], sandbox: false, script: 'return ["ERROR"]'],
script: [classpath: [], sandbox: false,
script: """
if params.ENVIRONMENT == 'lab'
return['aaa','bbb']
else
return ['ccc', 'ddd']
"""
]]]
])
])
pipeline {
agent any
tools {
maven 'Maven 3.6'
}
options {
disableConcurrentBuilds()
timestamps()
timeout(time: 30, unit: 'MINUTES')
ansiColor('xterm')
}
parameters {
choice(name: 'ENVIRONMENT', choices: "${environments}")
}
stages {
stage("Run Tests") {
steps {
sh "echo SUCCESS on ${params.ENVIRONMENT}"
}
}
}
}
But actually the second parameter is empty
Is it possible to use together scripted active choice parameter and declarative parameter?
UPD
Is there any way to pass list variable into script? For example
List<String> someList = ['ttt', 'yyyy']
...
script: [
classpath: [],
sandbox: true,
script: """
if (ENVIRONMENT == 'lab') {
return someList
}
else {
return['ccc', 'ddd']
}
""".stripIndent()
]
You need to use Active Choices Reactive Parameter which enable current job parameter to reference another job parameter value
environments = 'lab\nstage\npro'
properties([
parameters([
[$class: 'CascadeChoiceParameter',
choiceType: 'PT_SINGLE_SELECT',
description: 'Select a choice',
filterLength: 1,
filterable: true,
name: 'choice1',
referencedParameters: 'ENVIRONMENT',
script: [$class: 'GroovyScript',
fallbackScript: [
classpath: [],
sandbox: true,
script: 'return ["ERROR"]'
],
script: [
classpath: [],
sandbox: true,
script: """
if (ENVIRONMENT == 'lab') {
return['aaa','bbb']
}
else {
return['ccc', 'ddd']
}
""".stripIndent()
]
]
]
])
])
pipeline {
agent any
options {
disableConcurrentBuilds()
timestamps()
timeout(time: 30, unit: 'MINUTES')
ansiColor('xterm')
}
parameters {
choice(name: 'ENVIRONMENT', choices: "${environments}")
}
stages {
stage("Run Tests") {
steps {
sh "echo SUCCESS on ${params.ENVIRONMENT}"
}
}
}
}
As of Jenkins 2.249.2 without any plugin and using a declarative pipeline,
the following pattern prompt the user with a dynamic dropdown menu (for him to choose a branch):
(the surrounding withCredentials bloc is optional, required only if your script and jenkins configuratoin do use credentials)
node {
withCredentials([[$class: 'UsernamePasswordMultiBinding',
credentialsId: 'user-credential-in-gitlab',
usernameVariable: 'GIT_USERNAME',
passwordVariable: 'GITLAB_ACCESS_TOKEN']]) {
BRANCH_NAMES = sh (script: 'git ls-remote -h https://${GIT_USERNAME}:${GITLAB_ACCESS_TOKEN}#dns.name/gitlab/PROJS/PROJ.git | sed \'s/\\(.*\\)\\/\\(.*\\)/\\2/\' ', returnStdout:true).trim()
}
}
pipeline {
agent any
parameters {
choice(
name: 'BranchName',
choices: "${BRANCH_NAMES}",
description: 'to refresh the list, go to configure, disable "this build has parameters", launch build (without parameters)to reload the list and stop it, then launch it again (with parameters)'
)
}
stages {
stage("Run Tests") {
steps {
sh "echo SUCCESS on ${BranchName}"
}
}
}
}
The drawback is that one should refresh the jenkins configration and use a blank run for the list be refreshed using the script ...
Solution (not from me): This limitation can be made less anoying using an aditional parameters used to specifically refresh the values:
parameters {
booleanParam(name: 'REFRESH_BRANCHES', defaultValue: false, description: 'refresh BRANCH_NAMES branch list and launch no step')
}
then wihtin stage:
stage('a stage') {
when {
expression {
return ! params.REFRESH_BRANCHES.toBoolean()
}
}
...
}

Resources