I have a Jenkins pipeline that runs several stages in parallel. Some of those stages produce intermediate build files that I'd like to reuse in a later step:
pipeline {
stages {
stage("Parallel build") {
parallel {
stage("A") { /* produces file A */ }
stage("B") { /* produces file B */ }
stage("C") { /* produces nothing relevant */ }
}
}
stage("Combine") {
/* runs a task that needs files A and B */
}
}
}
As far as I've been able to tell, Jenkins will randomly give me the workspace from one of the parallel stages. So my Combine step will have file A, B or neither, but not both.
How do I resolve this issue?
Few ways to do this.
You can copy the files to a directory you desire(Which you know the path, you can create a sub directory with the build ID for it to be unique) and access them from there.
You can stash the files in the initial stages then unstash them and use them in the latter stages. Here is the documentation.
stash includes: 'something/A.txt', name: 'filea'
unstash 'filea'
Save the workspace location to a global variable and use it in the stages.
pipeline {
agent any
stages {
stage('Run Tests') {
parallel {
stage('Stage A') {
steps {
script {
sh '''
echo "STAGE AAAA"
pwd echo
"ATAGEA" > a.txt
'''
stageAWS = "$WORKSPACE"
}
}
}
stage('Stage B') {
steps {
script {
sh '''
echo "STAGE B"
pwd
'''
stageBWS = "$WORKSPACE" }
}
}
}
}
stage('Stage C') {
steps {
script { echo "$stageAWS" echo "$stageBWS" }
}
}
}
}
Related
I have some parallel stages in my Jenkins pipeline. They are all identical, except that they run on different agents:
stage {
parallel {
stage {
agent {
label 'agent-1'
}
steps {
sh 'do task number 468'
}
}
stage {
agent {
label 'agent-2'
}
steps {
sh 'do task number 468'
}
}
stage {
agent {
label 'agent-3'
}
steps {
sh 'do task number 468'
}
}
}
}
I want to add more parallel stages on more nodes, but the script is long and repetetive. What's the best way to rewrite this to tell jenkins to parallelize the same steps across agents 1, 2, 3, 4...etc?
Please see below code which will create and run the stage on multiple agents:
// Define your agents
def agents = ['agent-1','agent-2','agent-3']
def createStage(label) {
return {
stage("Runs on ${label}") {
node(label) {
// build steps that should happen on all nodes go here
echo "Running on ${label}"
sh 'do task number 468'
}
}
}
}
def parallelStagesMap = agents.collectEntries {
["${it}" : createStage(it)]
}
pipeline {
agent none
stages {
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
More information is available at : Jenkins examples
So, I want to run my parallel stages inside a stage but I also want to write some shared code by each parallel stage which I have written in steps of parallel parent stage
The problem I faced is that that the parallel stages are not being run
stages {
stage('partent stage 1'){
something here
}
stage('parent stage 2') {
steps {
// common code for parallel stages
parallel {
stage ('1'){
// some shell command
}
stage('2') {
// some shell command
}
}
}
}
}
For executing shared code you can define variables and functions outside of the declarative pipeline:
def foo = true
def checkFoo {
return foo
}
pipeline {
stage('parallel stage') {
parallel {
stage('stage 1') {
steps {
script {
def baz = checkFoo()
}
sh “echo ${baz}”
}
}
stage('stage 2') {
steps {
script {
def baz = checkFoo()
}
sh “echo ${baz}”
}
}
}
}
}
You can also write a shared library, which you can use in all or certain jobs.
I’ve deleted my first answer, since it was pure BS.
Ｉhave noticed that Jenkins pipeline file -- Jenkinsfile which have two syntax
Declarative
Scripted
I have made Declarative Script work to specify node to run my task. However I don't know how to modify my script to Scripted syntax.
My Declarative Script
pipeline {
agent none
stages {
stage('Build') {
agent { label 'my-label' }
steps {
echo 'Building..'
sh '''
'''
}
}
stage('Test') {
agent { label 'my-label' }
steps {
echo 'Testing..'
sh '''
'''
}
}
stage('Deploy') {
agent { label 'my-label' }
steps {
echo 'Deploying....'
sh '''
'''
}
}
}
}
I have tried to use in this way:
node('my-label') {
stage 'SCM'
git xxxx
stage 'Build'
sh ''' '''
}
But it seems Jenkins cannot find my node to run.
How about this simple example?
stage("one") {
node("linux") {
echo "One"
}
}
stage("two") {
node("linux") {
echo "two"
}
}
stage("three") {
node("linux") {
echo "three"
}
}
Or the below answer, this way you are guaranteed to have the stages run on the same node if there are multiple nodes with the same label and run interrupted by another job.
The above example will release the node after every stage, the below example will hold the node for all three stages.
node("linux") {
stage("one") {
echo "One"
}
stage("two") {
echo "two"
}
stage("three") {
echo "three"
}
}
I am attempting to write a scripted Jenkinsfile using the groovy DSL which will have parallel steps within a set of stages.
Here is my jenkinsfile:
node {
stage('Build') {
sh 'echo "Build stage"'
}
stage('API Integration Tests') {
parallel Database1APIIntegrationTest: {
try {
sh 'echo "Build Database1APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}, Database2APIIntegrationTest: {
try {
sh 'echo "Build Database2APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}, Database3APIIntegrationTest: {
try {
sh 'echo "Build Database3APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}
}
stage('System Tests') {
parallel Database1APIIntegrationTest: {
try {
sh 'echo "Build Database1APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}, Database2APIIntegrationTest: {
try {
sh 'echo "Build Database2APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}, Database3APIIntegrationTest: {
try {
sh 'echo "Build Database3APIIntegrationTest parallel stage"'
}
finally {
sh 'echo "Finished this stage"'
}
}
}
}
I want to have 3 stages: Build; Integration Tests and System Tests.
Within the two test stages, I want to have 3 sets of the tests executed in parallel, each one against a different database.
I have 3 available executors. One on the master, and 2 agents and I want each parallel step to run on any available executor.
What I've noticed is that after running my pipeline, I only see the 3 stages, each marked out as green. I don't want to have to view the logs for that stage to determine whether any of the parallel steps within that stage were successful/unstable/failed.
I want to be seeing the 3 steps within my test stages - marked as either green, yellow or red (Success, unstable or failed).
I've considered expanding the tests out into their own stages, but have realised that parallel stages are not supported (Does anyone know whether this will ever be supported?), so I cannot do this as the pipeline would take far too long to complete.
Any insight would be much appreciated, thanks
In Jenkins scripted pipeline, parallel(...) takes a Map describing each stage to be built. Therefore you can programatically construct your build stages up-front, a pattern which allows flexible serial/parallel switching.
I've used code similar to this where the prepareBuildStages returns a List of Maps, each List element is executed in sequence whilst the Map describes the parallel stages at that point.
// main script block
// could use eg. params.parallel build parameter to choose parallel/serial
def runParallel = true
def buildStages
node('master') {
stage('Initialise') {
// Set up List<Map<String,Closure>> describing the builds
buildStages = prepareBuildStages()
println("Initialised pipeline.")
}
for (builds in buildStages) {
if (runParallel) {
parallel(builds)
} else {
// run serially (nb. Map is unordered! )
for (build in builds.values()) {
build.call()
}
}
}
stage('Finish') {
println('Build complete.')
}
}
// Create List of build stages to suit
def prepareBuildStages() {
def buildStagesList = []
for (i=1; i<5; i++) {
def buildParallelMap = [:]
for (name in [ 'one', 'two', 'three' ] ) {
def n = "${name} ${i}"
buildParallelMap.put(n, prepareOneBuildStage(n))
}
buildStagesList.add(buildParallelMap)
}
return buildStagesList
}
def prepareOneBuildStage(String name) {
return {
stage("Build stage:${name}") {
println("Building ${name}")
sh(script:'sleep 5', returnStatus:true)
}
}
}
The resulting pipeline appears as:
There are certain restrictions on what can be nested within a parallel block, refer to the pipeline documentation for exact details. Unfortunately much of the reference seems biased towards declarative pipeline, despite it being rather less flexible than scripted (IMHO).
The pipeline examples page was the most helpful.
Here's a simple example without loops or functions based on #Ed Randall's post:
node('docker') {
stage('unit test') {
parallel([
hello: {
echo "hello"
},
world: {
echo "world"
}
])
}
stage('build') {
def stages = [:]
stages["mac"] = {
echo "build for mac"
}
stages["linux"] = {
echo "build for linux"
}
parallel(stages)
}
}
...which yields this:
Note that the values of the Map don't need to be stages. You can give the steps directly.
Here is an example from their docs:
Parallel execution
The example in the section above runs tests across two different platforms in a linear series. In practice, if the make check execution takes 30 minutes to complete, the "Test" stage would now take 60 minutes to complete!
Fortunately, Pipeline has built-in functionality for executing portions of Scripted Pipeline in parallel, implemented in the aptly named parallel step.
Refactoring the example above to use the parallel step:
// Jenkinsfile (Scripted Pipeline)
stage('Build') {
/* .. snip .. */
}
stage('Test') {
parallel linux: {
node('linux') {
checkout scm
try {
unstash 'app'
sh 'make check'
}
finally {
junit '**/target/*.xml'
}
}
},
windows: {
node('windows') {
/* .. snip .. */
}
}
}
To simplify the answer of #Ed Randall here.
Remember this is Jenkinsfile scripted (not declarative)
stage("Some Stage") {
// Stuff ...
}
stage("Parallel Work Stage") {
// Prealocate dict/map of branchstages
def branchedStages = [:]
// Loop through all parallel branched stage names
for (STAGE_NAME in ["Branch_1", "Branch_2", "Branch_3"]) {
// Define and add to stages dict/map of parallel branch stages
branchedStages["${STAGE_NAME}"] = {
stage("Parallel Branch Stage: ${STAGE_NAME}") {
// Parallel stage work here
sh "sleep 10"
}
}
}
// Execute the stages in parallel
parallel branchedStages
}
stage("Some Other Stage") {
// Other stuff ...
}
Please pay attention to the curly braces.
This will result in the following result (with the BlueOcean Jenkins Plugin):
I was also trying similar sort of steps to execute parallel stages and display all of them in a stage view. You should write a stage inside a parallel step as shown in the following code block.
// Jenkinsfile (Scripted Pipeline)
stage('Build') {
/* .. Your code/scripts .. */
}
stage('Test') {
parallel 'linux': {
stage('Linux') {
/* .. Your code/scripts .. */
}
}, 'windows': {
stage('Windows') {
/* .. Your code/scripts .. */
}
}
}
The above example with a FOR is wrong, as varible STAGE_NAME will be overwritten everytime, I had the same problem as Wei Huang.
Found the solution here:
https://www.convalesco.org/notes/2020/05/26/parallel-stages-in-jenkins-scripted-pipelines.html
def branchedStages = [:]
def STAGE_NAMES = ["Branch_1", "Branch_2", "Branch_3"]
STAGE_NAMES.each { STAGE_NAME ->
// Define and add to stages dict/map of parallel branch stages
branchedStages["${STAGE_NAME}"] = {
stage("Parallel Branch Stage: ${STAGE_NAME}") {
// Parallel stage work here
sh "sleep 10"
}
}
}
parallel branchedStages
I have used as below where the three stages are parallel.
def testCases() {
stage('Test Cases') {
def stages = [:] // declaring empty list
stages['Unit Testing'] = {
sh "echo Unit Testing completed"
}
stages['Integration Testing'] = {
sh "echo Integration Testing completed"
}
stages['Function Testing'] = {
sh "echo Function Testing completed"
}
parallel(stages) // declaring parallel stages
}
}
I have used stage{} in parallel blocks several times. Then each stage shows up in the Stage view. The parent stage that contains parallel doesn't include the timing for all the parallel stages, but each parallel stage shows up in stage view.
In blue ocean, the parallel stages appear separately instead of the stages showing. If there is a parent stage, it shows as the parent of the parallel stages.
If you don't have the same experience, maybe a plugin upgrade is due.
I'm using Jenkins Pipeline with the declarative syntax, currently with the following stages:
Prepare
Build (two parallel sets of steps)
Test (also two parallel sets of steps)
Ask if/where to deploy
Deploy
For steps 1, 2, 3, and 5 I need and agent (an executor) because they do actual work on the workspace. For step 4, I don't need one, and I would like to not block my available executors while waiting for user input. This seem to be referred to as either a "flyweight" or "lightweight" executor for the classic, scripted syntax, but I cannot find any information on how to achieve this with the declarative syntax.
So far I've tried:
Setting an agent directly in the pipeline options, and then setting agent none on the stage. This has no effect, and the pipeline runs as normalt, blocking the executor while waiting for input. It is also mentioned in the documentation that it will have no effect, but I thought I'd give it a shot anyway.
Setting agent none in the pipeline options, and then setting an agent for each stage except #4. Unfortunately, but expectedly, this allocates a new workspace for every stage, which in turn requires me to stash and unstash. This is both messy and gives me further problems in the parallel stages (2 and 3) because I cannot have code outside the parallel construct. I assume the parallel steps run in the same workspace, so stashing/unstashing in both would have unfortunate results.
Here is an outline of my Jenkinsfile:
pipeline {
agent {
label 'build-slave'
}
stages {
stage("Prepare build") {
steps {
// ...
}
}
stage("Build") {
steps {
parallel(
frontend: {
// ...
},
backend: {
// ...
}
)
}
}
stage("Test") {
steps {
parallel(
jslint: {
// ...
},
phpcs: {
// ...
},
)
}
post {
// ...
}
}
stage("Select deploy target") {
steps {
script {
// ... code that determines choiceParameterDefinition based on branch name ...
try {
timeout(time: 5, unit: 'MINUTES') {
deployEnvironment = input message: 'Deploy target', parameters: [choiceParameterDefinition]
}
} catch(ex) {
deployEnvironment = null
}
}
}
}
stage("Deploy") {
when {
expression {
return binding.variables.get("deployEnvironment")
}
}
steps {
// ...
}
}
}
post {
// ...
}
}
Am I missing something here, or is it just not possible in the current version?
Setting agent none at the top level, then agent { label 'foo' } on every stage, with agent none again on the input stage seems to work as expected for me.
i.e. Every stage that does some work runs on the same agent, while the input stage does not consume an executor on any agent.
pipeline {
agent none
stages {
stage("Prepare build") {
agent { label 'some-agent' }
steps {
echo "prepare: ${pwd()}"
}
}
stage("Build") {
agent { label 'some-agent' }
steps {
parallel(
frontend: {
echo "frontend: ${pwd()}"
},
backend: {
echo "backend: ${pwd()}"
}
)
}
}
stage("Test") {
agent { label 'some-agent' }
steps {
parallel(
jslint: {
echo "jslint: ${pwd()}"
},
phpcs: {
echo "phpcs: ${pwd()}"
},
)
}
}
stage("Select deploy target") {
agent none
steps {
input message: 'Deploy?'
}
}
stage("Deploy") {
agent { label 'some-agent' }
steps {
echo "deploy: ${pwd()}"
}
}
}
}
However, there are no guarantee that using the same agent label within a Pipeline will always end up using the same workspace, e.g. as another build of the same job while the first build is waiting on the input.
You would have to use stash after the build steps. As you note, this cannot be done normally with parallel at the moment, so you'd have to additionally use a script block, in order to write a snippet of Scripted Pipeline for the stashing/unstashing after/before the parallel steps.
There is a workaround to use the same build slave in the other stages.
You can set a variable with the node name and use it in the others.
ie:
pipeline {
agent none
stages {
stage('First Stage Gets Agent Dynamically') {
agent {
node {
label "some-agent"
}
}
steps {
echo "first stage running on ${NODE_NAME}"
script {
BUILD_AGENT = NODE_NAME
}
}
}
stage('Second Stage Setting Node by Name') {
agent {
node {
label "${BUILD_AGENT}"
}
}
steps {
echo "Second stage using ${NODE_NAME}"
}
}
}
}
As of today (2021), you can use nested stages (https://www.jenkins.io/doc/book/pipeline/syntax/#sequential-stages) to group all the stages that must run in the same workspace before the input step, and all the stages that must be run in the same workspace after the input step. Of course, you need to stash or to store artifacts in some external repository before the input step, because the second workspace may not be the same than the first one:
pipeline {
agent none
stages {
stage('Deployment to Preproduction') {
agent any
stages {
stage('Stage PRE.1') {
steps {
echo "StagePRE.1"
sleep(10)
}
}
stage('Stage PRE.2') {
steps {
echo "Stage PRE.2"
sleep(10)
}
}
}
}
stage('Stage Ask Deploy') {
steps {
input message: 'Deploy to production?'
}
}
stage('Deployment to Production') {
agent any
stages {
stage('Stage PRO.1') {
steps {
echo "Stage PRO.1"
sleep(10)
}
}
stage('Stage PRO.2') {
steps {
echo "Stage PRO.2"
sleep(10)
}
}
}
}
}
}