I running our maven project on a jenkins server with multiple stages inside the pipeline.
Every time I decide that the branch test does not need to continue and click on abort in the jenkins ui, I need to repeat this many times until the jenkins pipeline really stops.
I guess that our jenkinsfile does not really pick up that the job was aborted and I need to abort every stage to come to the end.
Is there a way to help jenkins to get out of the pipeline?
For example a variable I can check?
pipeline {
agent any
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
if (!currentBuild.isAborted) {
stage('Unit Tests') {
steps {
echo 'Unit Testing'
}
}
}
if (!currentBuild.isAborted) {
stage('Deploy') {
steps {
echo 'Deploying'
}
}
}
if (!currentBuild.isAborted) {
stage('Backend Integration Tests') {
steps {
echo 'Backend Int Tests'
}
}
}
if (!currentBuild.isAborted) {
stage('Frontend Integration Tests') {
steps {
echo 'Deploying....'
}
}
}
// done
}
}
I searched and experimented a lot, but couldn't find any working example of what I need.
I want to run 2 stages in parallel, each stage has several steps. And by parallel I mean the stages themselves and their respective steps. All existing examples and topics I found relate to executing steps in parallel inside a stage.
I don't need that, I need to run 2 stages alongside their steps in parallel. Please look at this screenshot:
Current behavior is that Parent Stage 1 is executed and then Parent Stage 2 starts to execute.
Is it possible to run "Parent Stage 1" and "Parent Stage 2" in parallel?
You can use stages inside the parallel block. e.g.
pipeline {
agent any
stages {
stage('Non Parallel Execution') {
steps {
sh 'echo step 1a'
sh 'echo step 2a'
sh 'echo step 3a'
}
}
stage('Parallel Execution') {
parallel {
stage('Parent A') {
steps {
sh 'echo step 1'
sh 'echo step 2'
}
}
stage('Parent B') {
steps {
sh 'echo step 3'
sh 'echo step 4'
}
}
}
}
}
}
src: https://jenkins.io/doc/book/pipeline/syntax/#parallel
If you're trying to have nested parallel execution instead, I'd suggest to read the answers to this question: Error: The parallel step can only be used as the only top-level step
The answer of #stefano is correct - use stages inside parallel although he didn't use it in his example. here is what I did:
pipeline {
agent { node {
label 'you node'
}}
stages {
stage ("Run Tests"){
parallel {
stage ("Run Parent 1 stages") {
stages {
stage("Parent 1 Stage 1") {
steps {
echo "Parent 1 stage 1"
}
}
stage("Parent 1 stage 2") {
steps {
echo "Parent 1 stage 2"
}
}
stage("Parent 1 stage 3") {
steps {
echo "Parent 1 stage 3"
}
}
}
}
stage ("Run Parent 2 stages") {
stages {
stage("Parent 2 stage 1") {
steps {
echo "Parent 2 stage 1"
}
}
}
}
}
}
}
}
and you will get this pipeline:
I am attempting to write a scripted Jenkinsfile using the groovy DSL which will have parallel steps within a set of stages.
Here is my jenkinsfile:
node {
stage('Build') {
sh 'echo "Build stage"'
}
stage('API Integration Tests') {
parallel Database1APIIntegrationTest: {
try {
sh 'echo "Build Database1APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}, Database2APIIntegrationTest: {
try {
sh 'echo "Build Database2APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}, Database3APIIntegrationTest: {
try {
sh 'echo "Build Database3APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}
}
stage('System Tests') {
parallel Database1APIIntegrationTest: {
try {
sh 'echo "Build Database1APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}, Database2APIIntegrationTest: {
try {
sh 'echo "Build Database2APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}, Database3APIIntegrationTest: {
try {
sh 'echo "Build Database3APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}
}
}
I want to have 3 stages: Build; Integration Tests and System Tests.
Within the two test stages, I want to have 3 sets of the tests executed in parallel, each one against a different database.
I have 3 available executors. One on the master, and 2 agents and I want each parallel step to run on any available executor.
What I've noticed is that after running my pipeline, I only see the 3 stages, each marked out as green. I don't want to have to view the logs for that stage to determine whether any of the parallel steps within that stage were successful/unstable/failed.
I want to be seeing the 3 steps within my test stages - marked as either green, yellow or red (Success, unstable or failed).
I've considered expanding the tests out into their own stages, but have realised that parallel stages are not supported (Does anyone know whether this will ever be supported?), so I cannot do this as the pipeline would take far too long to complete.
Any insight would be much appreciated, thanks
In Jenkins scripted pipeline, parallel(...) takes a Map describing each stage to be built. Therefore you can programatically construct your build stages up-front, a pattern which allows flexible serial/parallel switching.
I've used code similar to this where the prepareBuildStages returns a List of Maps, each List element is executed in sequence whilst the Map describes the parallel stages at that point.
// main script block
// could use eg. params.parallel build parameter to choose parallel/serial
def runParallel = true
def buildStages
node('master') {
stage('Initialise') {
// Set up List<Map<String,Closure>> describing the builds
buildStages = prepareBuildStages()
println("Initialised pipeline.")
}
for (builds in buildStages) {
if (runParallel) {
parallel(builds)
} else {
// run serially (nb. Map is unordered! )
for (build in builds.values()) {
build.call()
}
}
}
stage('Finish') {
println('Build complete.')
}
}
// Create List of build stages to suit
def prepareBuildStages() {
def buildStagesList = []
for (i=1; i<5; i++) {
def buildParallelMap = [:]
for (name in [ 'one', 'two', 'three' ] ) {
def n = "${name} ${i}"
buildParallelMap.put(n, prepareOneBuildStage(n))
}
buildStagesList.add(buildParallelMap)
}
return buildStagesList
}
def prepareOneBuildStage(String name) {
return {
stage("Build stage:${name}") {
println("Building ${name}")
sh(script:'sleep 5', returnStatus:true)
}
}
}
The resulting pipeline appears as:
There are certain restrictions on what can be nested within a parallel block, refer to the pipeline documentation for exact details. Unfortunately much of the reference seems biased towards declarative pipeline, despite it being rather less flexible than scripted (IMHO).
The pipeline examples page was the most helpful.
Here's a simple example without loops or functions based on #Ed Randall's post:
node('docker') {
stage('unit test') {
parallel([
hello: {
echo "hello"
},
world: {
echo "world"
}
])
}
stage('build') {
def stages = [:]
stages["mac"] = {
echo "build for mac"
}
stages["linux"] = {
echo "build for linux"
}
parallel(stages)
}
}
...which yields this:
Note that the values of the Map don't need to be stages. You can give the steps directly.
Here is an example from their docs:
Parallel execution
The example in the section above runs tests across two different platforms in a linear series. In practice, if the make check execution takes 30 minutes to complete, the "Test" stage would now take 60 minutes to complete!
Fortunately, Pipeline has built-in functionality for executing portions of Scripted Pipeline in parallel, implemented in the aptly named parallel step.
Refactoring the example above to use the parallel step:
// Jenkinsfile (Scripted Pipeline)
stage('Build') {
/* .. snip .. */
}
stage('Test') {
parallel linux: {
node('linux') {
checkout scm
try {
unstash 'app'
sh 'make check'
}
finally {
junit '**/target/*.xml'
}
}
},
windows: {
node('windows') {
/* .. snip .. */
}
}
}
To simplify the answer of #Ed Randall here.
Remember this is Jenkinsfile scripted (not declarative)
stage("Some Stage") {
// Stuff ...
}
stage("Parallel Work Stage") {
// Prealocate dict/map of branchstages
def branchedStages = [:]
// Loop through all parallel branched stage names
for (STAGE_NAME in ["Branch_1", "Branch_2", "Branch_3"]) {
// Define and add to stages dict/map of parallel branch stages
branchedStages["${STAGE_NAME}"] = {
stage("Parallel Branch Stage: ${STAGE_NAME}") {
// Parallel stage work here
sh "sleep 10"
}
}
}
// Execute the stages in parallel
parallel branchedStages
}
stage("Some Other Stage") {
// Other stuff ...
}
Please pay attention to the curly braces.
This will result in the following result (with the BlueOcean Jenkins Plugin):
I was also trying similar sort of steps to execute parallel stages and display all of them in a stage view. You should write a stage inside a parallel step as shown in the following code block.
// Jenkinsfile (Scripted Pipeline)
stage('Build') {
/* .. Your code/scripts .. */
}
stage('Test') {
parallel 'linux': {
stage('Linux') {
/* .. Your code/scripts .. */
}
}, 'windows': {
stage('Windows') {
/* .. Your code/scripts .. */
}
}
}
The above example with a FOR is wrong, as varible STAGE_NAME will be overwritten everytime, I had the same problem as Wei Huang.
Found the solution here:
https://www.convalesco.org/notes/2020/05/26/parallel-stages-in-jenkins-scripted-pipelines.html
def branchedStages = [:]
def STAGE_NAMES = ["Branch_1", "Branch_2", "Branch_3"]
STAGE_NAMES.each { STAGE_NAME ->
// Define and add to stages dict/map of parallel branch stages
branchedStages["${STAGE_NAME}"] = {
stage("Parallel Branch Stage: ${STAGE_NAME}") {
// Parallel stage work here
sh "sleep 10"
}
}
}
parallel branchedStages
I have used as below where the three stages are parallel.
def testCases() {
stage('Test Cases') {
def stages = [:] // declaring empty list
stages['Unit Testing'] = {
sh "echo Unit Testing completed"
}
stages['Integration Testing'] = {
sh "echo Integration Testing completed"
}
stages['Function Testing'] = {
sh "echo Function Testing completed"
}
parallel(stages) // declaring parallel stages
}
}
I have used stage{} in parallel blocks several times. Then each stage shows up in the Stage view. The parent stage that contains parallel doesn't include the timing for all the parallel stages, but each parallel stage shows up in stage view.
In blue ocean, the parallel stages appear separately instead of the stages showing. If there is a parent stage, it shows as the parent of the parallel stages.
If you don't have the same experience, maybe a plugin upgrade is due.
I am migrating a job from multijob to a Jenkins Declarative pipeline job. I am unable to run the parallel steps on multiple executors.
For example in the pipeline below, I see only one executor being used when I run the pipeline.
I was wondering why only a single executor is used. The idea is that each parallel step would be inoking a make target that would build a docker image.
pipeline {
agent none
stages {
stage('build libraries') {
agent { label 'master' }
steps {
parallel(
"nodejs_lib": {
dir(path: 'nodejs_lib') {
sh 'sleep 110'
}
},
"python_lib": {
dir(path: 'python_lib') {
sh 'sleep 100'
}
}
)
}
}
}
options {
ansiColor('gnome-terminal')
buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '', daysToKeepStr: '', numToKeepStr: '30'))
timestamps()
}
}
You can try the following way to perform parallel tasks execution for your pipeline job:
def tasks = [:]
tasks["TasK No.1"] = {
stage ("TASK1"){
node('master') {
sh '<docker_build_command_here>'
}
}
}
tasks["task No.2"] = {
stage ("TASK2"){
node('master') {
sh '<docker_build_command_here>'
}
}
}
tasks["task No.3"] = {
stage ("TASK3"){
node('remote_node') {
sh '<docker_build_command_here>'
}
}
}
parallel tasks
If you want to execute parallel tasks on a single node and also want to have the same workspace for both the tasks then you can go with the following approach:
node('master') {
def tasks = [:]
tasks["TasK No.1"] = {
stage ("TASK1"){
sh '<docker_build_command_here>'
}
}
tasks["task No.2"] = {
stage ("TASK2"){
sh '<docker_build_command_here>'
}
}
parallel tasks
}
I'm beginner to Jenkins. I have code pipeline structure like this
Repo1 -> Repo2 -> Repo3 -> Deploy
I already created such hierarchy via GUI but I want to create it via pipeline as code.I want to create chain of pipelines where I clone different repos and perform tests on it and then continue to another repo based on current pipeline post result.
This is my jenkinsfile - (psuedo code like as it gives me error to build)
pipeline {
agent any
stages {
stage('Build Repo1') {
steps {
sh 'echo "repo1 build!"'
}
}
stage('Test Repo1') {
steps {
sh 'echo "repo success!"'
}
}
}
post {
success {
pipeline {
agent any
stages {
stage('Build Repo2') {
steps {
sh 'echo "build repo2!"'
}
}
stage('Test Repo2') {
steps {
sh 'echo "test repo2!"'
}
}
}
post {
success {
# continue to generate pipeline for repo3
echo 'This will always run'
}
failure {
echo 'This will run only if failed'
}
}
}
}
failure {
echo 'This will run only if failed'
}
unstable {
echo 'This will run only if the run was marked as unstable'
}
changed {
echo 'This will run only if the state of the Pipeline has changed'
echo 'For example, if the Pipeline was previously failing but is now successful'
}
}
}
Please help!