I have around three Jenkins slave that are configured to run the same job allowing only one concurrent run on each slave. Each of these slave is connected to an embedded hardware that we run the job on. The total duration of the job is around 2 hours. The first 1 hour 50 mins is just taken to compile and configure the slave and the last 10 mins is when the embedded device is used. So basically I was looking for something that I can lock on for the last 10 mins. This would allow us to run multiple concurrent builds on the same slave.
Locks and Latches locks are shared across nodes.
What I am looking for is a node specific lock
If you can separate the problematic section from the compilation process you can just create another job to handle the last 10 minutes and call it using Parameterized Trigger Plugin. This job will run one instance at a time and will act as a native blocker for the run. That way, you can configure concurrent executions and throttling (if needed) on the main job, and create a "gate" to the problematic section.
Related
I have a multi-configuration job that uses a large amount of VMs for testing.
The Axis are something like:
30 VM slaves, 5 configurations, 5 different configurations
I would not like to run these sequentially, as the jobs would take forever. However, the default number of simultaneous runs is using up enough resources that I am getting random failures and disconnects.
Is there are way to specify the maximum number of simultaneous tests within this single running job?
I think you have to use a matrix job to trigger the builds of a separate job doing the real build. Then
you can use the Throttle Concurrent Builds Plugin to limit the number of parallel executions of that job you start by the matrix.
For multi project configuration
First you need to create a throttle category. In this case, the name is qa-aut and I limiting the number of execution to 2 for concurrent builds and concurrent builds per node. The node will have 4 executors available.
In your job configuration, make sure you don't run the multi-project sequentially:
Set up throttling builds, selecting "Throttle this project as part of one or more categories", "Multi-Project Throttle Category"(qa-aut) and "Throttle Matrix configuration builds". You can leave in blank the rest of the values
Make sure your node/master has enough executors available. In this case, the master will have available 4 executors
Execute your multi-project job
Instead of using 4 executors (all the availability), you will see it's using only 2 executors (2 threads) as specified in the category.
I manage a Jenkins server with a few hundred projects in the whole ecosystem. Many of the projects rely on upstream servers, that, unfortunately, are not always responsive. When I have a lag on these servers, my build queue can get to 10 or more. Is there a plugin or setting to send a warning email when the build queue exceeds a particular length?
I have been unable to find a plugin that does this, but you can query Jenkins for the information as detailed here: Jenkins command to get number of builds in queue.
If you have a Jenkins slave available you could set up a job that runs every 15 minutes and just hit each of the other Jenkins servers with the API call to get build queue counts (this is easy if you have just one master and many slaves.)
If you wanted to stay completely outside of Jenkins (not add another job to the mix) you could write a script to poll the Jenkins API for the information. You could then run that script under, say, a 15 minute (or some other relevant time step) timer using cron (or windows scheduled task). Admittedly then you have to dedicate some resources to running this job.
It looks like you could use python to get the build queue and check the length of the returned list. get_queue_info()
I haven't mucked about with the Jenkins API much myself so I'm not sure offhand exactly what the script would need, but it should be simple enough once you dig into it.
I've got two Jenkins jobs, a continuous integration job that runs whenever a new change is submitted, and a build/deploy job that runs everyday at noon. I do not want the builds to run simultaneously, so I am using the Throttle Concurrent Builds plugin to force only these two jobs to queue up if either of them is already running. Any of my other jobs are allowed to run at the same time as these two jobs.
I would like the build/deploy job to always run at noon, but sometimes a developer submits a change just before noon causing the build/deploy job to wait until the CI job finishes.
Is there a way to block only the CI job during a defined time frame, like 11:30am until 1pm, so that the build/deploy job will not be blocked at noon?
IMHO the are few possibilities:
a first pre-step which check the date and exit with error if the date do NOT match the requirements
two jobs which disable and enable the job programmatically calling the REST
eg job 1
curl -X POST http://usr:pwd#host/job/joobname/disable
and job 2
curl -X POST http://usr:pwd#host/job/joobname/enable
at given time.
You can try to add slave node which is offline from 11:30am until 1pm and restrict change-caused build to this.
Second possibility is to enable SCM poll schedule to check for changes during allowed time-frame.
I have a job that kicks off on any commit. It takes 5-10 minutes to run.
But if (say) 4 or 5 git commits come back-to-back I don't want 4 or 5 jobs run - just one job for the last commit. So basically if there is a job of type "X" in the build queue I don't want another job of type "X" in the queue.
That should be the default behavior if you're using the SCM trigger, default job parameters, and don't check the 'Execute concurrent builds if necessary' option.
First job is going to queue and run immediately.
On source change, next job is going to queue and wait until first one is complete.
A third SCM change would detect job already in queue and not do anything.
When first job is done, next one will start - and will use whatever is in the SCM at the moment it starts (not the moment it was scheduled).
That behavior can be changed using parameters, concurrent builds, job throttling, etc. My knowledge there might also be outdated (Jenkins is evolving pretty fast).
On a side note: multiple builds are not necessarily a bad thing - they give you failure locality, which might allow you faster identification of the offending commit. It doesn't matter much for 10 minutes builds, but if your build grows larger than that it can be a problem (with a large team, you can have a LOT of commits in 30 minutes).
Basically you just want to check if there is a new commit every 5 or 10 minutes? You can do that inside the triggering configuration: monitor source control every X minutes (CRON syntax: */15 * * * * for every 15mins)
If you check every 15 minutes if a new commit happened and your jobs only takes 10 minutes to run, there is no chance you would have another execution pending (unless someone ask for a "manual" construction...).
To avoid the latter case, you may consider the Throttle Concurrent Builds plugin
we are running Jenkins with lots of jobs. At the moment these jobs are kind of grouped by using "master jobs". These do nothing but start all jobs of one group. But, if one of these master jobs runs, it starts around 10 other jobs at one time. Depending on the duration of these jobs and the number of build processores (at the moment 6) Jenkins is blocked for a longer time (up to an hour). The other thing is, that these jobs are not really suitable for such massive parallelization.
To solve this, I'm looking for a way (a plugin), that allows to group some jobs and start them parallel, but limit the build processors used for the jobs of this group to a fixed number (e.g. 2). So it would be possible to run a group of jobs that compile java projects and parallel another group of jobs that installs test databases.
I tried the Build flow plugin, but it's not really the right one: you must separate the jobs manually to the sub-groups that run parallel and if a job in one sub-group failes, the following jobs of this group are not started.
So, maybe someone knows a Jenkins plugin that fits better? Thanks a lot in advance!
Frank
Throttle Concurrent Builds Plugin
Create some category my-group.
Add all the jobs into this group.
Set Maximum Total Concurrent Builds and Maximum Concurrent Builds Per Node.