I want to update my Jenkins pipeline in way that certain stages are only build when some specific files are changed (git is already integrated in the pipeline). I found a promising solution on this site, which would go like this for my use case (this code run successful):
stage("TEST STAGE 1") {
when {
anyOf { changeset "*dir1/*"; changeset "*dir2/*"; changeset "*somefile" }
}
steps {
// Do stuff
}
}
But I have more stages (TEST STAGE 2 and TEST STAGE 3) which should also be triggered only when these files are changed. To avoid writing the same code over and over (which would be bad practice), I implemented a function (I got the code from here):
def runStage(changeset) {
return {
changeset ==~ ("*dir1/*"||"*dir2/*"||"*somefile")
}
}
I call this function in the TEST stages (also TEST 2 and 3):
stage("TEST STAGE 1") {
when {
expression{ runStage(changeset) }
}
steps {
// Do stuff
}
}
But now my pipeline fails when entering the first TEST stage. I get this error:
hudson.remoting.ProxyException: groovy.lang.MissingPropertyException: No such property: changeset for class: WorkflowScript
Do you have an idea what I am doing wrong?
I found a solution.
This is my function:
def runStage() {
CHANGE_SET = sh (
script: 'git log -2 --name-only --oneline --pretty="format:"',
returnStdout: true
).trim()
echo "Current changeset: ${CHANGE_SET}"
return (CHANGE_SET ==~ "(.*)dir1(.*)|(.*)dir2(.*)|(.*)somefile")
}
I call it in my pipeline stage like this:
stage("TEST STAGE 1") {
when {
expression { runStage() }
}
steps {
//stuff
}
}
I would have prefered using changeset in the when block instead of git log, but it looks like it can't be done for my case.
I have a parallel stage setup, and would like to know if it's possible to run a script prior to the nested stages, so something like this:
stage('E2E-PR-CYPRESS') {
when {
allOf {
expression {
return fileExists("cypress.json")
}
branch "PR-*"
}
}
steps {
script {
stash name: 'cypress-dir', includes: 'cypress/**/*'
}
}
parallel {
stage('Cypress Tests 1') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
stage('Cypress Tests 2') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
}
post {
always {
e2eAfterCypressRun(this, true)
}
}
}
I know the above is wrong, I get the error Only one of "matrix", "parallel", "stages", or "steps" allowed for stage "E2E-PR-CYPRESS"
I already have the stash script in a setup stage at the beginning of my pipeline, but I'd like to be able to restart from this stage above on Jenkins, and so need the stash part in this stage as the parallel stages need to unstash the contents.
Updated Answer:
After playing a bit with the Restart from a Stage option there is seems to be a nice feature designed exactly for your needs called Preserving stashes for Use with Restarted Stages:
Normally, when you run the stash step in your Pipeline, the resulting
stash of artifacts is cleared when the Pipeline completes, regardless
of the result of the Pipeline. Since stash artifacts aren’t accessible
outside of the Pipeline run that created them, this has not created
any limitations on usage. But with Declarative stage restarting, you
may want to be able to unstash artifacts from a stage which ran before
the stage you’re restarting from.
To enable this, there is a job property that allows you to configure a
maximum number of completed runs whose stash artifacts should be
preserved for reuse in a restarted run. You can specify anywhere from
1 to 50 as the number of runs to preserve.
This job property can be configured in your Declarative Pipeline’s options section, as below:
options {
preserveStashes()
// or
preserveStashes(buildCount: 5)
}
This built in feature is exactly what you need to solve your issue without any special modifications to your code, as it will allow you to rerun the pipeline from any stage and still use the existing file that were previously stashed.
Original Answer:
You can actually achieve this quite simply using the scripted syntax for the parallel command, and it will also allow you to avoid the duplicate code in the parallel stages.
parallel: Execute in parallel
Takes a map from branch names to closures and an optional argument failFast which will terminate all branches upon a failure in any other branch:
parallel firstBranch: {
// do something
}, secondBranch: {
// do something else
},
failFast: true|false
In your case it can look like:
stage('E2E-PR-CYPRESS') {
when {
allOf {
expression {
return fileExists("cypress.json")
}
branch "PR-*"
}
}
steps {
script {
stash name: 'cypress-dir', includes: 'cypress/**/*'
// Define the parallel execution stages
def stages = ['Cypress Tests 1', 'Cypress Tests 2']
// Create the parallel executions and run them
parallel stages.collectEntries {
["Running ${it}": {
node('aws_micro_slave_e2e') {
skipDefaultCheckout()
runE2eTests()
}
}]
}
}
}
post {
always {
e2eAfterCypressRun(this, true)
}
}
}
This way you can easily add more parallel steps by updating the stages list, or even receive it as an input parameter. In addition you can create the parallel executions by different labels or tests suits, instead of the stage name.
You can add a Prepare stage at the top like this:
stages{
stage('Preperation'){
when {
allOf {
expression {
return fileExists("cypress.json")
}
branch "PR-*"
}
}
steps {
script {
stash name: 'cypress-dir', includes: 'cypress/**/*'
}
}
}
stage('E2E-PR-CYPRESS') {
parallel {
stage('Cypress Tests 1') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
stage('Cypress Tests 2') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
}
}
}
post {
always {
e2eAfterCypressRun(this, true)
}
}
An out of the box concept
Propose splitting the job into 2 parts taking the following into consideration:
Currently use an EC2 plugin, as the current agents are EC2
Running the parallel stages with the same stashed content ready to unstash
Create jenkins pipeline job 1:
This job will checkout the workspace with any type of agent
Create a packer json to create a customised AMI for the EC2
The customised AMI will stash the contents and move to a directory that will appear on the EC2 when the agent is built
Output the AMI ID, run a groovy job to update the EC2 plugin AMI ID with the customised AMI ID to temporarily set the AMI in memory on Jenkins
pipeline {
agent {
docker {
test-container
}
}
options {
buildDiscarder(
logRotator(
numToKeepStr: '10',
artifactNumToKeepStr: '10'
)
)
ansiColor('xterm')
gitConnection("git")
}
stages {
stage('Run Stash Cypress Functional Test') {
steps {
dir('functional-test') {
// develop branch is canary build, all other branches are stable builds
script {
sh """
# script to stash cypress tests
"""
}
}
}
}
stage('Functional Test AMI Build') {
steps {
dir('functional-test/packer') {
withAWS(role: 'PackerBuild', roleAccount: '123456789012', roleSessionName: 'Jenkins-Workflow-FunctionalTest-Packer') {
script {
sh """
# packer json script will require to copy contents from workspace, run the script to stash content
# packer json script will require to capture new AMI ID
# https://discuss.devopscube.com/t/how-to-get-the-ami-id-after-a-packer-build/36
# https://www.packer.io/docs/post-processors/manifest
packer validate FunctionalTestPacker.json
packer build -debug FunctionalTestPacker.json
# grab AMI ID and export as jenkins env variable
"""
}
}
}
}
}
stage('run groovy script to update AMI ID on EC2 plugin') {
steps {
dir(groovy job dir) {
script {
sh """
# run groovy job to update AMI on Jenkins EC2 plugin
# https://gist.github.com/vrivellino/97954495938e38421ba4504049fd44ea
"""
}
}
}
}
stage('Kickoff Functional Test Deploy') {
// pipeline checkbox parameter, when ticked it will automatically kick off the functional test pipeline
when {
expression {params.RUN_TESTS.toBoolean()}
}
steps {
script{
env.branch = params.BRANCH
sh """
echo "Branch is ${branch}"
"""
}
build job: 'workflow/CypressFunctionaTestDeployAndRun',
parameters: [
string(name: 'BRANCH', value: env.branch)
],
wait : false
}
}
}
post {
always {
cleanWs()
}
}
}
Create jenkins pipeline job 2:
This job will create the EC2 agents via the plugin from the customised AMI from pipeline job 1
This means your agents will have the same workspace ready to unstash - so you can execute a parallel run
Also you could move a lot of your user data script that is in the EC2 plugin as part of the customised AMI build, thus cut down the time for each EC2 agent to get ready to carry out execution
pipeline {
stages {
stage('E2E-PR-CYPRESS') {
when {
allOf {
expression {
return fileExists("cypress.json")
}
branch "PR-*"
}
}
}
parallel {
stage('Cypress Tests 1') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
stage('Cypress Tests 2') {
agent { label 'aws_micro_slave_e2e' }
options { skipDefaultCheckout() }
steps {
runE2eTests()
}
}
}
}
post {
always {
e2eAfterCypressRun(this, true)
}
}
}
I have a common Jenkins shared library for all the repositories as below.
vars/_publish.groovy
pipeline {
environment {
abc= credentials(’abc')
def= credentials(‘def’)
}
stages {
stage('Build') {
steps{
sh ‘docker build'
}
}
stage('Unit-test') {
steps{
sh ‘mvn test'
}
}
jenkinsfile
#Library('my-shared-library#branch') _
_publish() {
}
I have 10 Repository each has its own Jenkinsfile as shown above which refers to the jenkins shared library(vars/_publish.groovy). I have a condition here that I need to Pass. For few repository I want to skip the Unit test and just execute the build stage. For rest other repository I want both the stages. Is there anyone I can skip the particular stage based on the repository or repository name
Yes it's possible you can use when expression like this
pipeline {
agent any
stages {
stage('Test') {
when { expression { return repositoryName.contains('dev') } } <---------Add put your repository name 'dev' so whenever the repository names is ''dev' then execute this stage
steps {
script {
}
}
}
}
}
def repositoryName() {
def repositoryName = ['dev', 'test'] <----Add here the 10 repo name
return repositoryName
}
Here in my case repo names are dev and test so you can add yours accondigly
I would decorate my shared library and Jenkinsfile like this to achieve your scenario.
vars/_publish.groovy
def call(body={}) {
def pipelineParams = [:]
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = pipelineParams
body()
pipeline {
agent any;
stages {
stage('build') {
steps {
echo "BUILD"
}
}
stage('unitest') {
when {
anyOf {
equals expected: true, actual: pipelineParams.isEmpty();
equals expected: false, actual: pipelineParams.skipUnitest
}
}
steps {
echo "UNITEST"
}
}
}
}
}
I am enabling my shared library to accept parameter from Jenkinsfile and with when{} DSL deciding whether to skip unitest stage or not
Jenkinsfile
If your Jenkins file from the repo has below details, will skip the unitest stage
#Library('jenkins-shared-library')_
_publish(){
skipUnitest = true
}
below both scenario will run the unitest stage
#Library('jenkins-shared-library')_
_publish(){
skipUnitest = false
}
and
#Library('jenkins-shared-library')_
_publish(){
}
In a pipeline script of mine, I have the following:
pipeline {
agent any
options {
skipDefaultCheckout(true)
}
stages {
stage('Checkout') {
//Checkout steps
}
stage('Clean') {
when {
expression {
def result = input(message: 'Clean?', ok: 'Go for it!')
return true
}
beforeAgent true
}
parallel {
stage("Clean Project 1") {
// Clean steps
}
stage("Clean Project 2") {
// Clean steps
}
}
}
}
}
I'm expecting there to be a waiting step before the parallel stage "Clean" begins to run however the pipeline ends up looking like this:
I have even tried to add another step between checkout and clean with just an input as a step but that hasn't worked either. I could put the input st the end of the checkout steps but that is not desired.
My pipeline has a condition, wherein it runs the Node stage only if the branch is master. My problem is that the node:8 image is pulled by docker even if the stage is skipped. Is there a way to avoid this?
pipeline {
agent any
stages {
stage('Node') {
agent {
docker { image 'node:8' }
}
when {
branch 'master'
}
steps {
sh 'node -v'
}
}
stage('Maven') {
agent {
docker { image 'maven:3' }
}
steps {
sh 'mvn -v'
}
}
}
}
The when condition is evaluated on the agent. That's why the image is pulled. However, you can change this behaviour by using the beforeAgent option:
when {
beforeAgent true
branch 'master'
}
This will cause the when statement to be evaluated before entering the agent and should avoid pulling the image.