Is Jenkins capable of managing a large amount of jobs? More specifically, how powerful the Jenkins Master server have to be if it will be connected to, for example, 100 agents with each of them having 2 executors?
Although Jenkins Master is not actually running the real jobs but it still has to monitor the status of the agents and display console outputs, but I have no idea how resource-demanding these tasks are that's why I'm looking for some feedback.
Thanks :D
To make a right decision about resources and architecture you can read official documentation:
calculating
hardware recommendations
Or you can read whole article regarding your task
architecting for scale
Short atricle:
The equation for estimating the number of masters and executors needed
when the number of configured jobs is known is as follows:
masters = number of jobs/500
executors = number of jobs * 0.03
The equation for estimating the maximum number of jobs, masters, and
executors needed for an organization based on the number of developers
is as follows:
number of jobs = number of developers * 3.333
number of masters = number of jobs/500
number of executors = number of jobs * 0.03
Related
What is the impact on the running jobs if I change the number of executors in the production environment?
It depends on hardware used for master/slaves as well as on how much resource-intensive are your jobs.
We set the executors on our master to 0 and to our slaves - one executor per core.
Higher number of executors can result in degraded performance and higher flakiness of the jobs, especially true if they depend/share some data/application (database).
I am new to Jenkins and I have configured master-slave nodes as shown below, but I need help to configure the no of executors in each of the below slave nodes
Currently, I have configured 100 executors in each slave nodes
How many no of executors I can configure in each slave node and what fact(memory, RAM, etc) need to take consider when increasing the no of executors?
Maximum how many no of executors I can configure in each server?
Well it totally depends on your usage. There are multiple factors such as how much cpu and memory is available, how the build are going to execute and what kind of builds, how frequent these build should run etc. But I can clearly say that 100 is too big number. I would suggest go with 20 builds (if builds run frequently and have fair amount of CPU and memory) first and observe if is there any issue with numbers or not then you can increase accordingly.
here is very nice article check this out https://www.avantica.com/blog/jenkins-balance-load-master-slave-setup#:~:text=Jobs%20are%20built%20using%20executors,to%20build%20two%20different%20tasks.
I have a multi-configuration job that uses a large amount of VMs for testing.
The Axis are something like:
30 VM slaves, 5 configurations, 5 different configurations
I would not like to run these sequentially, as the jobs would take forever. However, the default number of simultaneous runs is using up enough resources that I am getting random failures and disconnects.
Is there are way to specify the maximum number of simultaneous tests within this single running job?
I think you have to use a matrix job to trigger the builds of a separate job doing the real build. Then
you can use the Throttle Concurrent Builds Plugin to limit the number of parallel executions of that job you start by the matrix.
For multi project configuration
First you need to create a throttle category. In this case, the name is qa-aut and I limiting the number of execution to 2 for concurrent builds and concurrent builds per node. The node will have 4 executors available.
In your job configuration, make sure you don't run the multi-project sequentially:
Set up throttling builds, selecting "Throttle this project as part of one or more categories", "Multi-Project Throttle Category"(qa-aut) and "Throttle Matrix configuration builds". You can leave in blank the rest of the values
Make sure your node/master has enough executors available. In this case, the master will have available 4 executors
Execute your multi-project job
Instead of using 4 executors (all the availability), you will see it's using only 2 executors (2 threads) as specified in the category.
If I have a 3 node cluster. I need to run a specific Quartz job as follows:
There is at a given time, many (say 30) of these jobs that need to be run.
Limit the number of a that Quartz job running on all clusters combined at the same time (to 10, because of system resources)
Limit the number of a that Quartz job running on a single server at the same time (to 5, because of CPU load)
How do I limit both the total number of simultaneous job instances to 10, and the number running on any one host to 5? Is this even possible?
Note that I cannot limit the number of threads as I have other jobs that need to run on the same servers at the same time, and those need threads as well.
Thanks.
While not exactly limiting the consecutive job count, you can limit the maximum thread count with the thread pool configuration. See Quartz Configuration Reference.
The Grails Quartz plugin comes with a handy script for installing the config file:
grails install-quartz-config
org.quartz.threadPool.threadCount
Can be any positive integer, although you should realize that only
numbers between 1 and 100 are very practical. This is the number of
threads that are available for concurrent execution of jobs. If you
only have a few jobs that fire a few times a day, then 1 thread is
plenty! If you have tens of thousands of jobs, with many firing every
minute, then you probably want a thread count more like 50 or 100
(this highly depends on the nature of the work that your jobs perform,
and your systems resources!).
I think that I found the answer.
The answer is to run two (or more) separate Quartz schedulers. A job in the first scheduler would schedule the job for the second scheduler, and the second would run them. The second scheduler could then be limited to (in this case) 5 threads, although the first scheduler could have more.
Some information about this can be found in
http://quartz-scheduler.org/documentation/quartz-2.2.x/cookbook/MultipleSchedulers
However I do not know how to implement two separate Quartz Schedulers in Grails. If anyone could help with that I would appreciate it. There is an existing Stack Overflow question about this though, although it is unanswered.
we are running Jenkins with lots of jobs. At the moment these jobs are kind of grouped by using "master jobs". These do nothing but start all jobs of one group. But, if one of these master jobs runs, it starts around 10 other jobs at one time. Depending on the duration of these jobs and the number of build processores (at the moment 6) Jenkins is blocked for a longer time (up to an hour). The other thing is, that these jobs are not really suitable for such massive parallelization.
To solve this, I'm looking for a way (a plugin), that allows to group some jobs and start them parallel, but limit the build processors used for the jobs of this group to a fixed number (e.g. 2). So it would be possible to run a group of jobs that compile java projects and parallel another group of jobs that installs test databases.
I tried the Build flow plugin, but it's not really the right one: you must separate the jobs manually to the sub-groups that run parallel and if a job in one sub-group failes, the following jobs of this group are not started.
So, maybe someone knows a Jenkins plugin that fits better? Thanks a lot in advance!
Frank
Throttle Concurrent Builds Plugin
Create some category my-group.
Add all the jobs into this group.
Set Maximum Total Concurrent Builds and Maximum Concurrent Builds Per Node.