I have a Jenkins jobs that runs as so (very simplified but the structure is there):
#!/usr/bin/env groovy
node('my_label'){
timestamps{
build()
postBuild()
}
}
def build(){
parallel{
flavorABuild: {
if(condtionA){
node(my_label){
stage("build flavor a"){
sh buildcommand
}
}
}
}
flavorBBuild: {
if(condtionB){
node(my_label){
stage("build flavor B"){
sh buildcommand
}
}
}
}
flavorCBuild: {
if(condtionC){
node(my_label){
stage("build flavor C"){
sh buildcommand
}
}
}
}
}
}
This works fine for my purposes as far as functionality goes, but when any of those conditions (build parameter check boxes) are not checked for building a specific flavor, that build still shows up in my BlueOcean view as a parallel build step just with no actions in it (automatically succeeded).
Is there a better/cleaner builtin way to generate conditional parallel builds? All suggestions welcome, however I am trying to avoid adding more plugins.
You can simply omit adding the stages to the parallel execution map. Refer to the following.
#!/usr/bin/env groovy
node('my_label'){
timestamps{
parallel getParallelStages()
}
}
def getParallelStages() {
def stageMap = [:]
if(ConditionA) {
stageMap['flavorABuild'] = { node(my_label){
stage("build flavor a"){
sh buildcommand
}
}
}
}
if(ConditionB) {
stageMap['flavorBBuild'] = { node(my_label){
stage("build flavor B"){
sh buildcommand
}
}
}
}
return stageMap
}
Related
I want to execute all the stages in parallel with the loop based on user input.
This gives error because script is not allowed under stages.
How should I achieve the same?
pipeline {
agent {
node {
label 'ec2'
}
}
stages{
script{
int[] array = params.elements;
for(int i in array) {
parallel{
stage('Preparation') {
echo 'Preparation'
println(i);
}
stage('Build') {
echo 'Build'
println(i);
}
}
}
}
}
}
If you are using declarative pipelines you have two options, first is to use static parallel stages which is an integral part of the declarative syntax but does not allow dynamic or runtime modifications.
The second option (which is probably what you attempted) is to use the scripted parallel function:
parallel firstBranch: {
// do something
}, secondBranch: {
// do something else
},
failFast: true|false```
When using it inside a declarative pipeline it should be used inside a script block like you did but the declarative basic directive must still be kept: pipeline -> stages -> stage -> steps -> script. In addition the scripted parallel function receives a specifically formatted map alike the example above.
In your case it can look somethong like:
pipeline {
agent {
node {
label 'ec2'
}
}
stages {
stage('Parallel Execution') {
steps {
script {
parallel params.elements.collectEntries {
// the key of each entry is the parallel execution branch name
// and the value of each entry is the code to execute
["Iteration for ${it}" : {
stage('Preparation') {
echo 'Preparation'
println(it);
}
stage('Build') {
echo 'Build'
println(it);
}
}]
}
}
}
}
}
}
Or if you want to use the for loop:
pipeline {
agent {
node {
label 'ec2'
}
}
stages {
stage('Parallel Execution') {
steps {
script {
map executions = [:]
for(int i in params.elements) {
executions["Iteration for ${it}" ] = {
stage('Preparation') {
echo 'Preparation'
println(i);
}
stage('Build') {
echo 'Build'
println(i);
}
}]
}
parallel executions
}
}
}
}
}
Other useful examples for the parallel function can be found here
I have the following two pipelines in my repository
#Field String ANDROID_EMULATOR = "android-emulator"
pipeline {
agent { label "android-emulator" }
stages {
stage("build") {
steps {
gradlew (":build")
}
}
}
}
void gradlew(String tasks) {
sh "./gradlew $tasks --profile"
}
#Field String ANDROID_EMULATOR = "android-emulator"
pipeline {
agent none
stages {
stage("PR checks") {
parallel {
stage("build 1") {
agent { label ANDROID_EMULATOR }
steps {
gradlew(":one:build")
}
}
stage("build 2") {
agent { label ANDROID_EMULATOR }
steps {
gradlew(":two:build")
}
}
}
}
}
}
void gradlew(String tasks) {
sh "./gradlew $tasks --profile"
}
As you can see, there is some code duplication between the two - ANDROID_EMULATOR and void gradlew(..).
I would like to move them into their own shared.groovy file:
#Field String ANDROID_EMULATOR = "android-emulator"
void gradlew(String tasks) {
sh "./gradlew $tasks --profile"
}
And be able to import it into my other pipelines with a single line of code. Gradle allows this to be done with apply('shared.groovy').
Jenkins seems to allow only shared libraries (which are global), and load statements (which need to be loaded as a part of a node, which does not scale well). Does Jenkins lack support for this basic style of code sharing here?
You can use the pipeline load which is more simple than using shared library, especially when you hope the shared.groovy in the same repo as your Jenkinsfiles.
// shared.groovy
def gradlew(String tasks) {
sh "./gradlew $tasks --profile"
}
return this // the return this must be have
// pipeline 1
pipeline {
agent { label "android-emulator" }
stages {
stage("build") {
steps {
scripts {
shared = load 'shared.groovy'
shared.gradlew (":build")
}
}
}
}
}
// pipeline 2
pipeline {
agent { label "android-emulator" }
stages {
stage("build") {
steps {
scripts {
shared = load 'shared.groovy'
shared.gradlew ("one:build")
}
}
}
}
}
Jenkins shared libraries have a well defined folder structure https://www.jenkins.io/doc/book/pipeline/shared-libraries/#directory-structure
You can try:
to implement this folder structure in a subfolder in your repo
to use a Dynamic Retrieval with a scm config that will check out the specific folder
I afraid it is too complicated and even not possible
I think the best approach is to create global shared libraries repo and to implement a gradleBuild custom step . In that case your code will be like
Pipeline 1:
#Library('somelib')
#Field String ANDROID_EMULATOR = "android-emulator"
pipeline {
agent { label "android-emulator" }
stages {
stage("build") {
steps {
gradleBuild ":build"
}
}
}
}
Pipeline 2:
#Library('somelib')
#Field String ANDROID_EMULATOR = "android-emulator"
pipeline {
agent none
stages {
stage("PR checks") {
parallel {
stage("build 1") {
agent { label ANDROID_EMULATOR }
steps {
gradlew(":one:build")
}
}
stage("build 2") {
agent { label ANDROID_EMULATOR }
steps {
gradlew(":two:build")
}
}
}
}
}
}
Shared libraries vars/gradeBuild.groovy file:
def call(String tasks) {
sh "./gradlew $tasks --profile"
}
I am trying to build a Jenkins pipeline which has a combination of parallel and sequential stages. I am able to accomplish the same with static data but failing to get it working when using dynamic data, i.e. when using a parameterized build and reading data from the build parameters.
Below snippet works fine
pipeline {
agent any
stages {
stage('Parallel Tests') {
parallel {
stage('Ordered Tests Set') {
stages {
stage('Building seq test 1') {
steps {
echo "build seq test 1"
}
}
stage('Building seq test 2') {
steps {
echo "build seq test 2"
}
}
}
}
stage('Building Parallel test 1') {
steps {
echo "Building Parallel test 1"
}
}
stage('Building Parallel test 2') {
steps {
echo "Building Parallel test 2"
}
}
}
}
}
}
Gives me the following execution result
Now i want to read the values from my build parameters and just loop the stages . This is what i have tried but could not get it to work. This bit of snippet is taken from another answer i found few months back in SO but unable to trace now, else would have added the link -
def parallelStagesMap = params['Parallel Job Set'].split(',').collectEntries {
["${it}" : generateStage(it)]
}
def orderedStagesMap = params['Ordered Job Set'].split(',').collectEntries {
["${it}" : generateStage(it)]
}
def orderedMap (){
def orderedStagesMapList= [:]
orderedStagesMapList['Ordered Tests Set']= {
stage('Ordered Tests Set') {
stages{
orderedStagesMap
}
}
}
return orderedStagesMapList;
}
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
}
}
}
pipeline {
agent none
stages {
stage ("Parallel Stage to trigger Tests"){
steps {
script {
parallel orderedMap()+parallelStagesMap
}
}
}
}
}
Declarative and Scripted Pipeline syntax do not mix in Pipeline, see Pipeline Syntax. Since you are dynamically creating a Pipeline definition based on the parameters, you should most likely go completely to Scripted Syntax, unless your use-case matches matrix.
Removing the Declarative syntax from your Pipeline Definition would give something like below. Note that I did not test it on the live Jenkins instance.
def parallelStagesMap = params['Parallel Job Set'].split(',').collectEntries {
["${it}" : generateStage(it)]
}
def orderedStagesMap = params['Ordered Job Set'].split(',').collectEntries {
["${it}" : generateStage(it)]
}
def orderedMap (){
def orderedStagesMapList= [:]
orderedStagesMapList['Ordered Tests Set']= {
stage('Ordered Tests Set') {
orderedStagesMap.each { key, value ->
value.call()
}
}
}
return orderedStagesMapList;
}
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
}
}
}
stage("Parallel Stage to trigger Tests") {
parallel orderedMap()+parallelStagesMap
}
We have a pipeline like this:
pipeline {
agent none
stages {
stage('Build') {
// ...
}
stage('Test') {
parallel {
stage('Test on Debian') {
agent {
label 'debian'
}
steps {
unstash 'compile-artifacts'
unstash 'dot-gradle'
sh './gradlew check --stacktrace'
}
post {
always {
junit '*/build/test-results/**/*.xml'
}
}
}
stage('Test on CentOS') {
agent {
label 'centos'
}
steps {
unstash 'compile-artifacts'
unstash 'dot-gradle'
sh './gradlew check --stacktrace'
}
post {
always {
junit '*/build/test-results/**/*.xml'
}
}
}
stage('Test on Windows') {
agent {
label 'windows'
}
steps {
unstash 'compile-artifacts'
unstash 'dot-gradle'
bat "gradlew.bat check --stacktrace"
}
post {
always {
junit '*/build/test-results/**/*.xml'
}
}
}
stage('Test on macOS') {
agent {
label 'macos'
}
steps {
unstash 'compile-artifacts'
unstash 'dot-gradle'
sh './gradlew check --stacktrace'
}
post {
always {
junit '*/build/test-results/**/*.xml'
}
}
}
}
}
}
}
Every stage is essentially identical, save for one line in the Windows block which I already know how to deal with, so is there a way to template out the common parts of these stages to remove the duplication?
I already tried putting a loop inline, but it's not something that declarative pipelines let you do. :(
You can refactor your step{}-blocks with groovy-methods:
def stageX(boolean linux) {
unstash 'compile-artifacts'
unstash 'dot-gradle'
if (linux) {
sh './gradlew check --stacktrace' }
else {
bat "gradlew.bat check --stacktrace" }
}
which you have to call like the following in your step{}:
steps {
script { stageX( true) } // or with false for your windows agent
}
Of course you can do the same for your junit-plugin-call:
def junitCall() {
junit '*/build/test-results/**/*.xml'
}
and call it like:
post {
always {
script { junitCall()
}
}
}
You won't win a lot of lines but it will improve the handling of the code a lot. If you want to cleanup your Jenkinsfile even more you could put the methods into a shared-library which you import so they aren't even declared in your Jenkinsfile.
Essentially what you want to do is currently not possible. As https://jenkins.io/doc/book/pipeline/shared-libraries/#defining-declarative-pipelines states:
Only entire pipelines can be defined in shared libraries as of this
time. This can only be done in vars/*.groovy, and only in a call
method. Only one Declarative Pipeline can be executed in a single
build, and if you attempt to execute a second one, your build will
fail as a result.
So you can define methods to bundle several steps or you can bundle a whole pipeline in a shared library but nothing in between. Which is a shame, really.
I can create pipelines by putting the following code into "Jenkinsfile" in my repository(called repo1) and creating a new item, through Jenkins GUI, to poll the repository.
pipeline {
agent {
docker {
image 'maven:3-alpine'
args '-v /root/.m2:/root/.m2'
}
}
stages {
stage('Build') {
steps {
sh 'mvn -B -DskipTests clean package'
}
}
stage('Test') {
steps {
sh 'mvn test'
}
post {
always {
junit 'target/surefire-reports/*.xml'
archiveArtifacts artifacts: 'target/*.jar', fingerprint: true
}
}
}
stage('Deploy') {
steps {
sh 'echo \'uploading artifacts to some repositories\''
}
}
}
}
But I have a case where I am not allowed create new items through Jenkins GUI but have a pre-defined job which reads JobDSL files in a repository I provide. So, I need to create the same pipeline through JobDSL but I cannot find the corresponding syntax for all the things, for instance, I couldn't find 'agent' DSL command.
Here is a job DSL code I was trying to change.
pipelineJob('the-same-pipeline') {
definition {
cps {
sandbox()
script("""
node {
stage('prepare') {
steps {
sh '''echo 'hello''''
}
}
}
""".stripIndent())
}
}
}
For instance, I could not find 'agent' command. Is it really possible to have the exact pipeline by using job DSL?
I found a way to create the pipeline item through jobDSL. So, the following jobDSL is creating another item which is just a pipeline.
pipelineJob('my-actual-pipeline') {
definition {
cpsScmFlowDefinition {
scm {
gitSCM {
userRemoteConfigs {
userRemoteConfig {
credentialsId('')
name('')
refspec('')
url('https://github.com/muatik/jenkins-as-code-example')
}
}
branches {
branchSpec {
name('*/master')
}
}
browser {
gitWeb {
repoUrl('')
}
}
gitTool('')
doGenerateSubmoduleConfigurations(false)
}
}
scriptPath('Jenkinsfile')
lightweight(true)
}
}
}
You can find the Jenkinsfile and my test repo here: https://github.com/muatik/jenkins-as-code-example