Anyone has idea how to limit number of concurrent builds on Jenkins for multi-branch pipeline?
I was searching out but almost every approach was about putting properties([disableConcurrentBuilds()]) which doesn't work in my case.
We use resource lock for unit tests, so they don't overlap. In my case no matter how many build executors I have they will just wait on lock to be released so they continue one by one while reserving build slot instead waiting in queue. I found some similar post:
Jenkins limit multibranch
for gitlab pipelines use parameter resource_group for create a resource group that ensures a job is mutually exclusive across different pipelines for the same project:
https://docs.gitlab.com/ee/ci/yaml/#resource_group
for jenkins: plugin and parameter "Block build if certain jobs are running"
https://plugins.jenkins.io/build-blocker-plugin/
Related
I'm using recent Jenkins version 2.286 and since this update there is an security hint: "You should set up distributed builds. Building on the controller node can be a security issue. See the documentation."
But I'm already doing so with three Jenkins nodes and I also fully understand the security implications.
The problem here is, that there are two jobs that need to run an the master, since they are the jobs that deploy those Jenkins nodes. That means I can not reduce the build executors to 0.
I've also tried using the Job Restrictions plugin to restrict which jobs can run on the master. This problem here is that all my jobs are waiting for the master queue do have a free slot available. I wonder why, because they all are declarative pipelines and define something like:
agent {
label 'some-different-node-label'
}
Which means they aren't really executed on the master node.
Questions here are:
Is this intentionally that all jobs require the master node before switching the agent?
Is there any configuration option to change that?
Is there a way to execute the deploy jobs on master, even if there aren't any executed defined (to bypass that behavior)?
Thanks.
With declarative pipelines the lightweight code checkout is done on the Master node to get a Jenkinsfile for that job. While this doesnt use an executor on the Master perhaps the Job Restriction Plugin is still blocking this (I havent used it before so cannot comment)
Also certain pipeline actions are delegated back to the Master node as well (e.g. the withAWSParameterStore step.
If you look at the console output for a Declarative pipeline job, you will see lots of output (mainly around library checkouts or git checkouts) before you see the start of the pipeline [Pipeline] Start of Pipeline. All that is done on the Master.
Unfortunately this cannot be changed as the Master needs to do this work to find out which agent type to delegate the job to.
Depending on how you are running you agents, you could use something like the EC2 Cloud Plugin to generate you agent nodes which wouldn't require a job to do it
We have a project where we have several Jenkins jobs:
One type of jobs that runs delivery (A),
one that does just compilation and unit tests (B)
and
one that runs integration tests, static code analysis et cetera (C).
We run on four Jenkins nodes (master + three slaves), and our jobs are a mix of declarative pipeline jobs and manually clicked in Jenkins-jobs.
We only want to run one integration test build per node at a time. However, we want to run as many deliveries (A) and code quality (B) builds as there are executors.
Until now, the Throttle concurrent builds (https://github.com/jenkinsci/throttle-concurrent-builds-plugin) plugin has satisfied our needs. However, this plugin does not support declarative pipeline builds, nor does it at all seem updated.
The Lockable resources plugin (https://github.com/jenkinsci/lockable-resources-plugin) seems promising, but we haven't found any way to lock the entire build with a resource-name dynamically set. That is, when we start a C build, we want it to lock "resource_{name of server}".
This plugin allows setting an entire-build-lock in the options directive,
× ut we haven't figured out how to evaluate an environment variable there.
Any suggestions would be highly appreciated!
So the workaround on our side was to rewrite the pipeline script from declarative to scripted syntax. Then, the throttle concurrent builds-plugin works as a charm.
When the bug https://issues.jenkins-ci.org/browse/JENKINS-49173 is fixed, the plugin will work with declarative pipelines as well.
I would like to build only one project at a time in entire jenkins server. I have tried Jenkins throttle concurrent plugin and lockable resource plugin but no luck. As lockable resource plugin doesn't give me option to lock in pipeline job.
I have 3 Jenkins pipeline jobs (job have pipeline script):
JOB1
JOB2
JOB3
which has some common thing at beginning of the job (clearing content).
Running one by one manually doesn't have any problem if job completed but if JOB1 is building and JOB2 starts in between then it interrupts to JOB1 and build fails for JOB1.
Even when I start jobs using CLI, you never know which job might be running. So, I'm looking for solution to block JOBY if JOBX is running (X, Y can be 1,2 or 3) and allow only one job to run in entire Jenkins server. Like I said, throttle concurrent plugin gives customization option only for respective job instead of for multiple jobs?
Can anyone suggest some solution for multi pipeline jobs block to run only once?
Install Build Blocker Plugin
In configuration of JOBY check "Block build if certain jobs are running".
Put JOBX name in Blocking Jobs text area, each job on new line.
Note, you can also use regex to define in single line jobs having the same prefix but ending with different numbers.
I'm trying to find a way to ensure that an entire pipeline completes on a specific executor without allowing other jobs to be run on that executor:
my pipeline essentially looks like:
Build -> Deploy -> API testing -> selenium testing
as we have multiple teams, all running multiple paralell pipelines i want to ensure that on a per slave basis all builds complete in the pipeline before any others begin
Is anyone aware of a plugin that does this?
You can look into the Locks and Latches plugin that can help you enforce this by assigning a lock to the relevant pipeline.
I have never tried your setup, but it might work.
Also, consider restricting the number of executors on the slave to 1, so only a single pipeline can "fit in".
I hope this helps.
I have a Jenkins server with half a dozen builds. Each of these builds is composed of a parent job that triggers anywhere between 4 and 6 smaller jobs that run in parallel. I use the EC2 plugin to keep the number of active slaves in line with the number of queued builds. Or in other words, slaves are coming and going all the time. Each slave has 7 executors (parent job + max(4, 6)).
It is absolutely crucial that all jobs of a build are executed on the same machine. I also cannot allow any jobs from build A to execute on a machine that has jobs from build B running.
What I'm looking for is a way that prevents Jenkins from using any inactive executors of a node as long as any jobs from a previous build are still active on it.
I've spent the day experimenting with a combination of the Throttle Concurrent Builds Plugin and the NodeLabel Parameter Plugin. Unfortunately, there seems to be a bug somewhere that causes throttled builds to not contribute to the Load Statistics of a slave. This in turn means that throttled build will never trigger Jenkins to spin up additional slaves. Obviously this is totally unacceptable.
You can try and use "This build is parameterized"
and pass the $NODE_NAME as a parameter between the builds and then use it at the "Restrict where this project can be run"