Trigger multiple builds parallel at one fire in Jenkins Pipeline - jenkins

Context
I have a few Jenkins pipelines which are based on the regex to select the matching jobs to run. The number of downstream jobs for each pipeline around 60.
So I have written down the declarative scripts to select these matching job then built it with Jenkins Build Plugin parallel. In general, it's something as below:
Finish deploy to A --> Pipeline A (master node)
->> 60 downstream jobs (slave nodes)
Finish deploy to B --> Pipeline B (master node)
->> 60 downstream jobs (slave nodes)
Finish deploy to C --> Pipeline C (master node)
->> 60 downstream jobs (slave nodes)
The master node is the server used to run Jenkins.
The slave nodes are the AWS EC2 instances used to run the task, my pool around 32 servers which can be used up to 180 tasks at once.
Expected
Let's say I have those pipeline triggered pipeline A, pipeline B, pipeline C in sequence then I expect the downstream jobs are triggered gonna be in the queue in sequence. It means first 60 jobs from A will be scheduled, built then so on pipeline B, pipeline C, given the slave executors are still free and available to use.
Observed
Even the AWS EC2 instances running no task (all available), the pipeline did not trigger all the jobs at one fire but part of its. It means a random number of jobs in pipeline A will be built first, then after a while, the rest will be built.
The pipeline script:
Stage('Integration Test Run') {
steps {
script {
matchingJobs = commonPipelineMethods.getTestJobs(venture_to_test, testAgainst)
parallel matchingJobs.collectEntries{downstreamJob-> [downstreamJob.name, commonPipelineMethods.buildSingleJob(this, downstreamJob)]}
}
}
}
def buildSingleJob(steps, downstreamJob) {
return {
def result = steps.build job: downstreamJob.fullName, propagate: false
steps.echo "${downstreamJob.fullName} finished: ${result.rawBuild.result}"
}
}
So I'm not sure if I need anything to config, setting up the pipeline script to get those downstream job running at one fire.
Could anybody please look into this and give me some suggestion? Thanks

Related

Concatenated jobs in Jenkins

I have to solve the following problem:
I have a job A in Jenkis. In one of the stages another Job B is started by the "job build" command.
For Job A to finish, it needs to wait for Job B to finish.
The problem is that Job B does not start because it is waiting for Job A to finish.
I'm using pipeline script in all Jobs.
I imagined that using the command " build job: 'my Job', propagate: true, wait: true" Jenkins would start Job B and after completion of B, it would return to complete Job A
workflow example
I'm assuming you just have one executor configured in your Jenkins Server/Slave. Hence Multiple builds are unable to run at the same time, which causes your deadlock.
In order to increase the executor count follow the following instructions. If you are just using the master to build. Go to Dashboard > Manage Jenkins > Configure and increase the executor count.
If you have slaves go to Dashboard > Manage Jenkins > Manage nodes and clouds click on the edit icon on the Slave and increase the executor count(more than 1) for slaves.

Jenkins Pipeline Jobs, keep them in queue until an executor is free

I have a Jenkins pipeline job, which in turn starts dynamically several other pipeline jobs in parallel. At the moment I have only 1 node to execute these jobs.
All of these jobs start immediately. One of them actually does and starts its stage, and the other ones show in the log that they waiting for an executor:
Started by upstream project "parent project" build number 278
originally caused by:
Started by user user
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Still waiting to schedule task
Waiting for next available executor on win
I'd very much like to change that behavior and have them start when an executor is free. Until then they should stay in the build queue. The reason is that the durations will get shown all wrong, and in the stage view they all start showing "almost complete" when they haven't actually started.
This is the start of the child projects pipeline script, I do require a matching agent immediately:
pipeline {
agent {
label "win"
}
stages {
stage('[Jenkins] Setup') {
Is this achievable somehow? Would I need to switch to scripted pipeline maybe?
You can use throttle concurrent builds plugin and configure the properties "Maximum Total Concurrent Builds" in your job to required number. This way it will prevent the builds from running if it exceeds than the number you configured and you can see the builds in queue.

Jenkins Pipeline Plugin execute two pipelines in parallel and make downstream pipeline wait

We are using Jenkins and the Pipeline plugin for CI/CD. We have two pipelines that we need to run parallel, and there is a downstream pipeline which should trigger ONLY if the two upstream pipelines both finish and are successful.
P1--
| -- P3
P2--
Basically P3 should run only when P1 and P2 are finished and successful and not depend on just one of them.
Is there a way to achieve this? We are using 2.5 version of the plugin.
Since stages only run if previous stages run successfully, and since you can execute other pipelines via build, and since there is a magical instruction called parallel, I think this might do it:
pipeline {
agent { label 'docker' }
stages {
stage("build_p1_and_p2_in_parallel") {
steps {
parallel p1: {
build 'p1'
}, p2: {
build 'p2'
}
}
}
stage("build_p3_if_p1_and_p2_succeeded") {
steps {
build 'p3'
}
}
}
}
Use the "Snippet Generator" embedded in your jenkins instance to figure out what the argument to build should be. If it's another pipeline at the same level as the top level Jenkinsfile, you could just reference it by job name. Caveat: I've used parallel, but never build within parallel, but it seems like it should work.
You can try and wrap the pipelines jobs with MultiJob plugin that can implement the logic that you require as 2 jobs inside a phase.

how to trigger jenkins downstream job only both upstreams jobs successfully executed

I have created Job A which looks for upstream jobs (Job B and Job C) success result and triggers a shell script to verify a condition.
Once Job B and Job C executed successfully Job A executes downstream jobs (Job D and Job E).
I have used reverse (to configure upstream jobs) and downstream-ext (to configure downstream jobs) plugins in Job A using JJB.
Issue I am facing here is: After Job B is executed successfully without waiting for Job C result. Job A should wait for both Job B and Job C and then execute based on the result.
Could you please help me how to configure this scenario.
You can try using the Join Plugin, here is the documentation:
https://wiki.jenkins-ci.org/display/JENKINS/Join+Plugin
This'd be easier if you convert your A job to a build flow https://wiki.jenkins.io/display/JENKINS/Build+Flow+Plugin?focusedCommentId=60917290 or even better it's successor Pipeline 2.0 https://jenkins.io/doc/book/pipeline/
(Groovy) Code in A would then be something like:
if (build('scenario-B-Job') && build('scenario-C-Job')) {
build('scenario-E-Job')
build('scenario-D-Job')
}
You can also parallelize (B,C and then D,E) to shorten overall execution times if you have enough slaves around.

Jenkins jobs on slave servers

I have many Jenkins Jobs that I need to run on every Build,
At present time I have 4 slave servers.
I would like the jobs to run in parallel as much as possible, hence I defined the jobs as follow:
Execute concurrent builds if necessary - Disabled
Restrict where this project can be run - Enabled with the following values SalveLinux1HT||SalveLinux2HT||SalveLinux3HT||SalveLinux4HT
To my understanding if Job A and B are triggered at the same time, one should use 1HT and the other should use 2HT and they can run in parallel
however Jenkins build job A on all 4 slaves and only after it's finished he will build job B on all 4 slaves
This is the opposite of my goal
Any ideas?
Thanks in advance
You can use
Build Flow Plugin
You can find both installation and configuration instructions of this plugin at the above mentioned link.
If you want to run any jobs in parallel you can use following scripts:
parallel (
// job A and B will be scheduled in parallel.
{ build("jobA") },
{ build("jobB") }
)
// jobC will be triggered after jobs A and B are completed
build("jobC")

Resources