is it possible to manage some resources in Jenkins? By that I mean only certain number of jobs should run in parallel. E.g. my license server can provide only 4 licenses, so each job that uses such license should first check if some counter is greater that zero, next decrement and increment back after license is not needed anymore. There is Exclusion Plugin, but I think it can only create mutex (counter equals 1) not semaphore (counter equals 1 or more). Simple variable in script is not a solution, because same counter must be visible by master and all slaves.
Any ideas how to handle that?
Thanks in advance.
You can use the Throttle Concurrent Builds Plugins. You can do it in two ways:
If the restriction applies to only one job, you can edit the job itself. There is a Throttle Concurrent Builds setting, where you can configure how many concurrent builds of the same job are allowed.
If there are multiple jobs that have to be restricted, then you can define Multi-Project Throttle Categories in the global configuration. You define a category with a restriction, and then tag every job that has to be restricted with this category.
Related
I want to run one job multiple times (each build with different parameters) on 2 executors.
I want to execute them based on their build priority value.
Unfortunately Priority Sorter plugin doesn't help in my case, it doesn't sort it correctly - my builds are being executed based on the timestamp they were added to the queue instead of priority.*
I believe this priority mechanism should be implemented before, on a queue level.
How to achieve that?
*-I tested it on the newest Jenkins version and the newest Priority Sorter plugin version
I think what you should try is Accelerated Build Now plugin. This plugin allows the Jenkins users to launch a build right there, even if the queue is long. It prioritises human launched jobs and brings them to the top.
The Priority Sorter I am using (v3.6.0 and Jenkins v.2.73.3) will not see a job unless you have enabled the Execute Concurrent builds if necessary.
So, allow parallel builds for that job and maybe decrease the number of cores to 1. See if that works. If not, you can try Throttle Concurrent Builds Plugin. This allows you to assign as many cores as you want in the specific job.
Here is a patch for Priority Queue plugin. For your case it could be this patch.
I have a multi-configuration job that uses a large amount of VMs for testing.
The Axis are something like:
30 VM slaves, 5 configurations, 5 different configurations
I would not like to run these sequentially, as the jobs would take forever. However, the default number of simultaneous runs is using up enough resources that I am getting random failures and disconnects.
Is there are way to specify the maximum number of simultaneous tests within this single running job?
I think you have to use a matrix job to trigger the builds of a separate job doing the real build. Then
you can use the Throttle Concurrent Builds Plugin to limit the number of parallel executions of that job you start by the matrix.
For multi project configuration
First you need to create a throttle category. In this case, the name is qa-aut and I limiting the number of execution to 2 for concurrent builds and concurrent builds per node. The node will have 4 executors available.
In your job configuration, make sure you don't run the multi-project sequentially:
Set up throttling builds, selecting "Throttle this project as part of one or more categories", "Multi-Project Throttle Category"(qa-aut) and "Throttle Matrix configuration builds". You can leave in blank the rest of the values
Make sure your node/master has enough executors available. In this case, the master will have available 4 executors
Execute your multi-project job
Instead of using 4 executors (all the availability), you will see it's using only 2 executors (2 threads) as specified in the category.
In my newly installed Jenkins, I have four jobs. I can only run two concurrently. If I trigger the build of a third job, it is set in the queue and triggered once one of the first two finishes.
I know my server can handle more than two concurrent jobs at a time. How can I increase this default threshold of two?
If it means anything, these are not build-a-deployable package kind of jobs but environment prep jobs that instantiate various DBs. So the jobs simply invoke a python script on the Jenkins server, which is the same script across multiple jobs but each job invokes it with different input params. The jobs are 100% independent of one another and do not share any resource except the script.
You go to Manage Jenkins --> Configure System, then change # of executors:
I have in Jenkins a set of jobs A1, A2, ... that can be executed concurrently, as well as a job B that must never be executed concurrently with any job Ai. All these jobs run on the same node (the jobs Ai use a pool of executors that, for reasons that can't be helped, occasionally have to be shepherded by job B). Can I enforce this in Jenkins?
The concept is similar to that of a shared mutex; the jobs Ai require shared-level access to the pool, while the job B requires exclusive-level access.
I'm looking at the Throttle Concurrent Builds plugin, but it appears from the options that it provides that it only has one level of access. I could make B never be concurrent with any Ai, but only by making all Ai mutually exclusive as well.
Is there a way to achieve shared-mutex-like behavior, either with this plugin or otherwise?
There's the Block queued job plugin:
Plugin for blocking/unblocking job in queue by some conditions configured in this job.
There's the Build Blocker Plugin:
This plugin keeps the actual job in the queue if at least one name of currently running jobs is matching with one of the given regular expressions.
Is it possible to block a certain jenkins job while another one is running as some sort of condition?
I would like job "A" to be disabled while job "B" is running, and once job "B" is done, then job "A" should be available again.
I have read that it is possible to block upstream jobs when the jobs are part of a flow and the flow is running, but I would like to know if it s possible for 2 completely independent jobs.
Use this Build Blocker Jenkins Plugin.
You can block A in job B's config. As long as 'A' is running, 'B' would not run.
You can block Both A & B in B's config, thus B would run only no other As or Bs are running.
Additionally, You can block B in A's config, they will block each other.
*job names are case sensitivity
You can use the Throttle Concurrent Builds Plugin to do this.
You're going to want to:
create a category for the two jobs in the Jenkins global
configuration.
In each build check the Throttle Concurrent Builds
option.
Choose Throttle this project as part of one or more
categories.
Set Maximum Total Concurrent Builds (and Maximum
Concurrent Builds per Node if applicable) to 1
Check the category box to mark the job as part of the category.
This will limit either job from running while the other one runs, so if Job A is running then B won't start, and if Job B is running than A won't start.