Declarative Pipeline When Conditions - jenkins

I'm in need of help with the when condition on Jenkins. I want to take the words FULL_BUILD from the merge request title and then execute different stages based on whether it is a FULL_BUILD or someone just submitting a merge request to master that doesn't need to go through Veracode, SonarQube, etc etc (these stages are not pasted in as they are just repeats of the when conditions below). However, I have to repeat this crazy when condition on EVERY stage, as well as sometimes creating a special stage that only executes on FULL_BUILD or "regular" builds.
Has anyone created a #NonCPS script to set a true/false variable? Or is there a crafty way to execute the script on startup to set a reusable variable?
I want to have the users execute everything from their Gitlab MR and not have to go to Jenkins to hit a button (hence I do not use a parameter of boolean).
pipeline {
agent {
node {
label 'master'
}
}
parameters{
//I am trying to pull the information for a full build from the Merge Request
}
environment {
//Assume random variables all work fine
}
options {
skipDefaultCheckout()
gitLabConnection('GitLab_Generic')
timeout(time: 60, unit: 'MINUTES')
}
triggers {
gitlab(triggerOnPush: true, triggerOnMergeRequest: true, branchFilterType: 'All')
}
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Build') {
steps {
dir("${env.WORKSPACE}\\${params.PROJECT_NAME}") {
bat 'nuget.exe restore %PROJECT_NAME%.sln'
}
dir("${env.WORKSPACE}\\${params.PROJECT_NAME}\\build") {
bat "\"${tool 'msbuild'}\" %COMPONENT%.XML /p:Configuration=Debug "
}
dir("${env.WORKSPACE}") {
echo "Creating a Build status file"
writeFile file: "output/MR_Title.txt", text: "BUILD STATUS:"
}
}
}
stage('Check MR FULL_BUILD' ){
when {
branch 'master'
}
steps{
dir("${env.WORKSPACE}") {
//writeFile file: "MR_Title.txt", text: "BUILD STATUS:"
powershell '& "./build/scripts/MergeRequestAPI.ps1" -GIT_CREDENTIALS $env:GIT_API_TOKEN -PROJECT_ID $env:GIT_PROJECT_ID | Out-File output/MR_Title.txt -Encoding utf8"'
}
}
}
stage('Package Snapshot') {
when {
allOf {
branch 'master'
not {
expression {
return readFile('output/MR_Title.txt').contains("FULL BUILD")
}
}
}
}
steps {
dir("${env.WORKSPACE}\\${params.PROJECT_NAME}\\build") {
bat "\"${tool 'msbuild'}\" %COMPONENT%.XML /t:Publish /p:version=${env.SnapshotComponentVersion} "
}
}
}
stage('Package Full Build') {
when {
allOf {
branch 'master'
expression {
return readFile('output/MR_Title.txt').contains("FULL BUILD")
}
}
}
steps {
dir("${env.WORKSPACE}\\${params.PROJECT_NAME}\\build") {
bat "\"${tool 'msbuild'}\" %COMPONENT%.XML /t:Publish /p:version=${env.ComponentVersion} "
}
}
}
}
}

Related

How do I run the same stages on multiple nodes in Jenkins declarative pipeline?

We have a pipeline like this:
pipeline {
agent none
stages {
stage('Build') {
// ...
}
stage('Test') {
parallel {
stage('Test on Debian') {
agent {
label 'debian'
}
steps {
unstash 'compile-artifacts'
unstash 'dot-gradle'
sh './gradlew check --stacktrace'
}
post {
always {
junit '*/build/test-results/**/*.xml'
}
}
}
stage('Test on CentOS') {
agent {
label 'centos'
}
steps {
unstash 'compile-artifacts'
unstash 'dot-gradle'
sh './gradlew check --stacktrace'
}
post {
always {
junit '*/build/test-results/**/*.xml'
}
}
}
stage('Test on Windows') {
agent {
label 'windows'
}
steps {
unstash 'compile-artifacts'
unstash 'dot-gradle'
bat "gradlew.bat check --stacktrace"
}
post {
always {
junit '*/build/test-results/**/*.xml'
}
}
}
stage('Test on macOS') {
agent {
label 'macos'
}
steps {
unstash 'compile-artifacts'
unstash 'dot-gradle'
sh './gradlew check --stacktrace'
}
post {
always {
junit '*/build/test-results/**/*.xml'
}
}
}
}
}
}
}
Every stage is essentially identical, save for one line in the Windows block which I already know how to deal with, so is there a way to template out the common parts of these stages to remove the duplication?
I already tried putting a loop inline, but it's not something that declarative pipelines let you do. :(
You can refactor your step{}-blocks with groovy-methods:
def stageX(boolean linux) {
unstash 'compile-artifacts'
unstash 'dot-gradle'
if (linux) {
sh './gradlew check --stacktrace' }
else {
bat "gradlew.bat check --stacktrace" }
}
which you have to call like the following in your step{}:
steps {
script { stageX( true) } // or with false for your windows agent
}
Of course you can do the same for your junit-plugin-call:
def junitCall() {
junit '*/build/test-results/**/*.xml'
}
and call it like:
post {
always {
script { junitCall()
}
}
}
You won't win a lot of lines but it will improve the handling of the code a lot. If you want to cleanup your Jenkinsfile even more you could put the methods into a shared-library which you import so they aren't even declared in your Jenkinsfile.
Essentially what you want to do is currently not possible. As https://jenkins.io/doc/book/pipeline/shared-libraries/#defining-declarative-pipelines states:
Only entire pipelines can be defined in shared libraries as of this
time. This can only be done in vars/*.groovy, and only in a call
method. Only one Declarative Pipeline can be executed in a single
build, and if you attempt to execute a second one, your build will
fail as a result.
So you can define methods to bundle several steps or you can bundle a whole pipeline in a shared library but nothing in between. Which is a shame, really.

Multiple Jenkins Nodes in Pipeline

Currently we have a jenkins pipeline with 4 stages. Setup, Build, Deploy, Teardown. Deploy and Teardown prompt for manual user input. Because of this, we don`t want manual user input to take up an executor. So, we want to use agent none. However, when resuming, there is no guarentee we get the same jenkins workspace. Stash/unstash says it uses alot of resources, so if you have large files not to use it. Is there a way to get the exact slave, and when resuming, run back on that same slave?
I have something like this now I also tried agent gcp at top level, and putting agent none in manual input
pipeline {
agent none
environment {
userInput = false
}
stages {
stage('Setup') {
agent { node { label 'gcp' } }
steps {
deleteDir()
dir('pipelines') {
checkout scm
}
dir('deployment_pipelines'){
git branch: __deployment_scripts_code_branch, credentialsId: 'jenkins', url: __deployment_scripts_code_repo
}
dir('gcp_template_core'){
git branch: __gcp_template_code_branch, credentialsId: 'jenkins', url: __gcp_template_code_repo
}
dir('control_repo'){
git branch: _control_repo_branch, credentialsId: 'jenkins', url: _control_repo
}
// Copy core templates to the project
sh('bash deployment_pipelines/deployment/setup.sh gcp_template_core/gcp_foundation/ control_repo')
}
}
stage('Build') {
agent { node { label 'gcp' } }
steps {
sh('printenv') //TODO: Remove. Debug only
sh('python deployment_pipelines/deployment/build.py control_repo --env ${_env_type_long}')
}
}
stage('Deploy') {
agent { node { label 'gcp' } }
steps {
sh('python deployment_pipelines/deployment/deploy.py control_repo --env ${_env_type_short}')
}
}
stage('Release') {
steps {
agent none
script {
sh('python deployment_pipelines/deployment/set_manual_approvers.py deployment_pipelines/config/production-release-approvers.yaml -o approver.txt')
def approvers = readFile('approver.txt')
try {
userInput = input(
message: 'Do you want to proceed with Release?',
submitter: approvers)
} catch(err) { // input false
//def user = err.getCauses()[0].getUser() //need script approval for getUser()
userInput = false
// echo "Aborted by [${user}]"
}
agent { node { label 'gcp' } }
if(userInput)
{
sh("echo 'Do Release'")
}
}
}
}
stage('Teardown'){
agent { node { label 'gcp' } }
steps {
script {
def approvers = readFile('approver.txt')
try {
userInput = input(
message: 'Do you want to proceed with Teardown?',
submitter: approvers)
} catch(err) { // input false
//def user = err.getCauses()[0].getUser() //need script approval for getUser()
userInput = false
// echo "Aborted by [${user}]"
}
if(userInput)
{
sh("echo 'Do Teardown'")
}
}
}
}
}
post {
always {
echo 'DO TEARDOWN REGARDLESS'
}
}
}
agent none should be above step block in stage('Release'). You can refer https://jenkins.io/doc/book/pipeline/syntax/#agent for syntax and flow

Jenkins Multibranch job with declarative pipeline cloning repo for every stage

Trying to create a workflow in Jenkins using Declarative Pipeline to do something like this:
Checkout the code on 'master'
Build solution on 'master' (I know this is not a secure way to do it, but Jenkins is in the intranet so it should be fine for us)
Stash artifacts (.dll, .exe, .pdb, etc) => 1st stage
Unstash artifacts on nodes depending on what it's needed (Unit tests on a slave, Integration tests on another one and Selenium tests on a another one) => 2nd stage
Run tests depending on the slave => 3rd stage running in parallel
The problem that I'm facing is that the git checkout (GitSCM) is executed for every stage.
My pipeline looks like this:
pipeline {
agent {
label {
label "master"
customWorkspace "C:\\Jenkins\\workspace\\CustomWorkspace"
}
}
options {
timestamps()
}
stages {
stage("Build") {
agent {
label {
label "master"
customWorkspace "C:\\Jenkins\\workspace\\CustomWorkspace"
}
}
steps {
/*
steps to build the solution here
*/
//Sleep because stashing fails otherwise
script {
sleep(1)
}
dir("${env.WORKSPACE}\\UnitTests\\bin\\Release") {
stash name: 'unit-tests'
}
dir("${env.WORKSPACE}\\WebUnitTests\\bin\\x64\\Release") {
stash name: 'web-unit-tests'
}
}
stage('Export artefacts') {
agent {
label {
label "UnitTest"
customWorkspace "C:\\Jenkins\\workspace\\CustomWorkspace"
}
}
steps {
echo "Copying dlls from master to ${env.NODE_NAME}"
dir("${env.WORKSPACE}\\UnitTests\\bin\\Release") {
unstash 'unit-tests'
}
}
}
stage('Run tests') {
parallel {
stage("Run tests #1") {
agent {
label {
label "UnitTest"
customWorkspace "C:\\Jenkins\\workspace\\CustomWorkspace"
}
}
steps {
/*
run tests here
*/
}
post {
//post results here
}
}
//other parallel stages
}
}
}
}
So, as mentioned earlier, the GitSCM (code checkout) is a part of and performed for every stage:
Build stage
Export stage
A couple simple changes should solve this. You need to tell the pipeline script not to checkout by default every time a node is allocated. Then you need to tell it to do the checkout where you need it:
pipeline {
agent {
label {
label "master"
customWorkspace "C:\\Jenkins\\workspace\\CustomWorkspace"
}
}
options {
timestamps()
skipDefaultCheckout() // Don't checkout automatically
}
stages {
stage("Build") {
agent {
label {
label "master"
customWorkspace "C:\\Jenkins\\workspace\\CustomWorkspace"
}
}
steps {
checkout scm //this will checkout the appropriate commit in this stage
/*
steps to build the solution here
*/
//Sleep because stashing fails otherwise
script {
sleep(1)
}
dir("${env.WORKSPACE}\\UnitTests\\bin\\Release") {
stash name: 'unit-tests'
}
dir("${env.WORKSPACE}\\WebUnitTests\\bin\\x64\\Release") {
stash name: 'web-unit-tests'
}
}
stage('Export artefacts') {
agent {
label {
label "UnitTest"
customWorkspace "C:\\Jenkins\\workspace\\CustomWorkspace"
}
}
steps {
echo "Copying dlls from master to ${env.NODE_NAME}"
dir("${env.WORKSPACE}\\UnitTests\\bin\\Release") {
unstash 'unit-tests'
}
}
}
stage('Run tests') {
parallel {
stage("Run tests #1") {
agent {
label {
label "UnitTest"
customWorkspace "C:\\Jenkins\\workspace\\CustomWorkspace"
}
}
steps {
/*
run tests here
*/
}
post {
//post results here
}
}
//other parallel stages
}
}
}
I have added 2 lines there. One in the options section (skipDefaultCheckout()), and a checkout scm in the first stage.

How to use Jenkins declarative pipeline to build and test on multiple platforms

I'm trying to do something that I feel should be simple to do, but I can't figure out how.
Basically I have a Jenkins master (running on Linux) and two slaves, one on Windows and the other on macOS.
I want to build my project on all 3 platforms and run GTest tests on all 3 platforms too.
I can build and run the test, but the junit step doesn't seem to collect any test results.
I tried to put the post block everywhere, but it just doesn't work. If I try to put the post block in the Test stage or as a sibling of stages, I get the following error:
Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node
which is caused by agent none - the post block doesn't know where to run.
So I tried to put the post block inside the node block in my parallel step for the Test stage, but it doesn't seem to do anything - it doesn't even show up in the console output.
Here's my Jenkinsfile:
pipeline {
agent none
stages {
stage ('Clean') {
steps {
parallel (
"linux" : {
node ("linux") {
dir("build") {
deleteDir()
writeFile file:'dummy', text:'' // Creates the directory
}
}
},
"windows" : {
node('windows') {
dir("build") {
deleteDir()
writeFile file:'dummy', text:'' // Creates the directory
}
}
},
"mac" : {
node('mac') {
dir("build") {
deleteDir()
writeFile file:'dummy', text:'' // Creates the directory
}
}
}
)
}
}
stage ('Build') {
steps {
parallel (
"linux" : {
node ("linux") {
checkout scm
dir("build") {
sh '/opt/cmake/bin/cmake .. -DCMAKE_BUILD_TYPE=Release'
sh 'make'
}
}
},
"windows" : {
node('windows') {
checkout(changelog: false, scm: scm) // Changelog to false, otherwise Jenkins shows duplicates. Only linux (the Jenkins master) has the changelog enabled.
dir("build") {
bat 'cmake .. -G "Visual Studio 15 2017 Win64" -DCMAKE_PREFIX_PATH=C:/Qt/5.9.1/msvc2017_64'
bat "\"${tool 'MSBuild'}\" project.sln /p:Configuration=Release /p:Platform=\"x64\" /p:ProductVersion=1.0.0.${env.BUILD_NUMBER} /m"
}
}
},
"mac" : {
node('mac') {
checkout(changelog: false, scm: scm) // Changelog to false, otherwise Jenkins shows duplicates. Only linux (the Jenkins master) has the changelog enabled.
dir("build") {
sh 'cmake .. -DCMAKE_PREFIX_PATH=/usr/local/Cellar/qt/5.9.1 -DCMAKE_BUILD_TYPE=Release'
sh 'make'
}
}
}
)
}
}
stage ('Test') {
steps {
parallel (
"linux" : {
node ("linux") {
dir('Build') {
sh './bin/project-tests --gtest_output=xml:project-tests-results.xml'
// Add other test executables here.
}
post {
always {
junit '*-tests-results.xml'
}
}
}
},
"windows" : {
node('windows') {
dir("build") {
bat 'tests\\project\\Release\\project-tests --gtest_output=xml:project-tests-results.xml'
// Add other test executables here.
}
post {
always {
junit '*-tests-results.xml'
}
}
}
},
"mac" : {
node('mac') {
dir("build") {
sh './bin/project-tests --gtest_output=xml:project-tests-results.xml'
// Add other test executables here.
}
post {
always {
junit '*-tests-results.xml'
}
}
}
}
)
}
}
}
}
What am I doing wrong?
post{} block should only follow steps{} or parallel{} (for parallel stages) to take effect.
If you require post to be executed in a node environment, you should provide a node to the entire stage (agent{} statement).
You could try to use parallel stages execution. Also I'd suggest to use functions to shorten the code.
Something like this:
void Clean() {
dir("build") {
deleteDir()
writeFile file:'dummy', text:'' // Creates the directory
}
}
void SmthElse(def optionalParams) {
// some actions here
}
pipeline {
agent none
options {
skipDefaultCheckout(true) // to avoid force checkouts on every node in a first stage
disableConcurrentBuilds() // to avoid concurrent builds on same nodes
}
stages {
stage('Clean') {
failfast false
parallel {
stage('Linux') {
agent {label 'linux'}
steps {Clean()}
post {
// post statements for 'linux' node
SmthElse(someParameter)
}
}
stage('Windows') {
agent {label 'windows'}
steps {Clean()}
post {
// post statements for 'windows' node
}
}
stage('MacOS') {
agent {label 'mac'}
steps {Clean()}
post {
// post statements for 'mac' node
}
}
}
post {
// Post statements OUTSIDE of nodes (i.e. send e-mail of a stage completion)
}
}
// other stages (Build/Test/Etc.)
}
}
Alternatively you can use node in post statements:
stage('Test') {
steps {
// your parallel Test steps
}
post {
always {
script {
parallel (
"linux" : {
node('linux') {
// 'linux' node post steps
}
},
"windows" : {
node('windows') {
// 'windows' node post steps
}
}
// etc
)
}
}
}
}

Use a lightweight executor for a declarative pipeline stage (agent none)

I'm using Jenkins Pipeline with the declarative syntax, currently with the following stages:
Prepare
Build (two parallel sets of steps)
Test (also two parallel sets of steps)
Ask if/where to deploy
Deploy
For steps 1, 2, 3, and 5 I need and agent (an executor) because they do actual work on the workspace. For step 4, I don't need one, and I would like to not block my available executors while waiting for user input. This seem to be referred to as either a "flyweight" or "lightweight" executor for the classic, scripted syntax, but I cannot find any information on how to achieve this with the declarative syntax.
So far I've tried:
Setting an agent directly in the pipeline options, and then setting agent none on the stage. This has no effect, and the pipeline runs as normalt, blocking the executor while waiting for input. It is also mentioned in the documentation that it will have no effect, but I thought I'd give it a shot anyway.
Setting agent none in the pipeline options, and then setting an agent for each stage except #4. Unfortunately, but expectedly, this allocates a new workspace for every stage, which in turn requires me to stash and unstash. This is both messy and gives me further problems in the parallel stages (2 and 3) because I cannot have code outside the parallel construct. I assume the parallel steps run in the same workspace, so stashing/unstashing in both would have unfortunate results.
Here is an outline of my Jenkinsfile:
pipeline {
agent {
label 'build-slave'
}
stages {
stage("Prepare build") {
steps {
// ...
}
}
stage("Build") {
steps {
parallel(
frontend: {
// ...
},
backend: {
// ...
}
)
}
}
stage("Test") {
steps {
parallel(
jslint: {
// ...
},
phpcs: {
// ...
},
)
}
post {
// ...
}
}
stage("Select deploy target") {
steps {
script {
// ... code that determines choiceParameterDefinition based on branch name ...
try {
timeout(time: 5, unit: 'MINUTES') {
deployEnvironment = input message: 'Deploy target', parameters: [choiceParameterDefinition]
}
} catch(ex) {
deployEnvironment = null
}
}
}
}
stage("Deploy") {
when {
expression {
return binding.variables.get("deployEnvironment")
}
}
steps {
// ...
}
}
}
post {
// ...
}
}
Am I missing something here, or is it just not possible in the current version?
Setting agent none at the top level, then agent { label 'foo' } on every stage, with agent none again on the input stage seems to work as expected for me.
i.e. Every stage that does some work runs on the same agent, while the input stage does not consume an executor on any agent.
pipeline {
agent none
stages {
stage("Prepare build") {
agent { label 'some-agent' }
steps {
echo "prepare: ${pwd()}"
}
}
stage("Build") {
agent { label 'some-agent' }
steps {
parallel(
frontend: {
echo "frontend: ${pwd()}"
},
backend: {
echo "backend: ${pwd()}"
}
)
}
}
stage("Test") {
agent { label 'some-agent' }
steps {
parallel(
jslint: {
echo "jslint: ${pwd()}"
},
phpcs: {
echo "phpcs: ${pwd()}"
},
)
}
}
stage("Select deploy target") {
agent none
steps {
input message: 'Deploy?'
}
}
stage("Deploy") {
agent { label 'some-agent' }
steps {
echo "deploy: ${pwd()}"
}
}
}
}
However, there are no guarantee that using the same agent label within a Pipeline will always end up using the same workspace, e.g. as another build of the same job while the first build is waiting on the input.
You would have to use stash after the build steps. As you note, this cannot be done normally with parallel at the moment, so you'd have to additionally use a script block, in order to write a snippet of Scripted Pipeline for the stashing/unstashing after/before the parallel steps.
There is a workaround to use the same build slave in the other stages.
You can set a variable with the node name and use it in the others.
ie:
pipeline {
agent none
stages {
stage('First Stage Gets Agent Dynamically') {
agent {
node {
label "some-agent"
}
}
steps {
echo "first stage running on ${NODE_NAME}"
script {
BUILD_AGENT = NODE_NAME
}
}
}
stage('Second Stage Setting Node by Name') {
agent {
node {
label "${BUILD_AGENT}"
}
}
steps {
echo "Second stage using ${NODE_NAME}"
}
}
}
}
As of today (2021), you can use nested stages (https://www.jenkins.io/doc/book/pipeline/syntax/#sequential-stages) to group all the stages that must run in the same workspace before the input step, and all the stages that must be run in the same workspace after the input step. Of course, you need to stash or to store artifacts in some external repository before the input step, because the second workspace may not be the same than the first one:
pipeline {
agent none
stages {
stage('Deployment to Preproduction') {
agent any
stages {
stage('Stage PRE.1') {
steps {
echo "StagePRE.1"
sleep(10)
}
}
stage('Stage PRE.2') {
steps {
echo "Stage PRE.2"
sleep(10)
}
}
}
}
stage('Stage Ask Deploy') {
steps {
input message: 'Deploy to production?'
}
}
stage('Deployment to Production') {
agent any
stages {
stage('Stage PRO.1') {
steps {
echo "Stage PRO.1"
sleep(10)
}
}
stage('Stage PRO.2') {
steps {
echo "Stage PRO.2"
sleep(10)
}
}
}
}
}
}

Resources