This may be a crazy idea but I'm just throwing it.
Is it possible to have one Jenkins master's executors available as slave(s) (executors) from another Jenkins Master?
I.e. Let's say JenkinsMaster1 (has 10 executors). It has bunch of slaves (in various OS with various # of executors per slave) but all of them are used/running something.
There's another JenkinsMaster2 and this instance has the same setup (bunch of slaves with N no. of executors) but this one has some/a lot of free executors (on master or it's slaves).
The question is NOT, why I can't just create a new slave for JenkinsMaster1 if I need a job configured in JenkinsMaster1 instance to run (while every other executor in JenkinsMaster1/its slave are in use) or why not add more/increase executors of JenkinsMaster1 master/slaves BUT how can/is it even possible to use JenkinsMaster2's executors (or it's slaves i.e. owned by JenkinsMaster2) to run a job which is configured on JenkinsMaster1.
Related
I have master jenkins and slave jenkins. I hav kept slave jenkins no of build executors as 1. Slave Jenkins also has 1 pipeline (Lets say pipeline A).
lets suppose a job from slave jenkins' own pipeline is running right now (Job A). I schedule a job from the master jenkins for slave jenkins (Job B).
I dont want Job B to run while job A is running as both jobs use shared resources.
Right now, Job B runs in parallel with Job A, which is causing Job A to fail.
How to do that?
Thanks!
Your implementation is a bit tricky since you are talking about 2 separate machines with 2 separate Jenkins instances. One option is to get rid of the Jenkins instance in the slave machine and move the Jenkins job that runs on it to the master machine. Then, you can schedule the job to use the resources of the slave machine while being managed by the master machine. If you do that, no further configuration will be needed since you have set the number of executors to 1.
If that is not possible, the other option is to find a way for them to communicate with each other that a build is running. Consider the third point of this answer. You can have a variable in a database somewhere and when one job starts, it updates the variable. Before the second job starts, it has to poll the variable to see if there is a job already running. If yes, the build doesn't start, if no, build starts and updates the variable.
Another less elegant solution is to simply have a text file in a location accessible to both machines and write the variable data into that instead of a database.
One way to do this is by using the Lockable Resources Plugin.
Currently, we have two machines. One has Jenkins installed and is hosted as master in Jenkins and another one is Slave. Number of executors for both Nodes are set to 1.
I am not exactly sure how Jenkins work behind the scenes but currently when I triggered 2 build jobs simultaneously, it somehow runs only on slave node (and put another build job in queue), if I disconnect the slave and leave only master, then it would run on master(and put another build job in queue).
How to configure Jenkins so that it leverage all my available nodes (master and slave). In other words, I would like to have all available nodes consumes the queue and not just for one of the Nodes.
As I understand, you need to enable Execute concurrent builds if necessary option in your job configuration and then you will be able to run your job simultaneously on all available nodes.
In addition to the above answer. We can also restrict the job to a particular node on which it should run.
For eg
A setup of 3 servers(2 Linux and one windows )
1 Linux server acts as master
1 Linux server acts as node
1 window server as as node
If we have a job that needs to be run on the windows node you can go to the job configuration and restrict the job to run on that node using the node name or label.
Additionally, the no. of executes define the instances of the slave or master node that can be executed parallelly across different jobs.
For running same job you need to check the enable concurrent build option and assign a label having more than 1 nodes in it
Cheers,
Yash
I am curios if it's possible to have one computer/node that would be shared between two jenkins instances?
I know that one can make two same copies of node configurations on Jenkins instances and change workspace, but I am aiming for this scenario.
Jenkins A with Node A
Jenkins B with Node A
Jenkins A starts job on Node A
Jenkins B puts Node A offline
Jenkins B starts job on Node A
Jenkins A puts Node A offline
Node A can only be used once at a time, and there should not be parallel jobs running no matter of different workspaces.
I know that it can be achieved when you have one jenkins instance, but how to do it if you have to use two of them at least?
NOTE: both Jenkins instances are located on different machines.
I believe jenkins mesos plugin is what you needed.
Use following steps to install & configure:
step1
step2
step3
step4
Of course, you had to install a mesos cluster for yourself before do above, see mesos official site. This may cost some time if you are not familar with mesos.
And in step4, you'd better set slave cpus equal with the number of your real slave node.
The principle is:
Mesos first sends resource offer to Jenkins Master A, then jenkins master A will launch tasks on the slave node A, as the resource was used by jenkins master A now. Resource offer will not be sent to Jenkins Master B again, so Jenkins Master B is waiting.
After Jenkins Master A finish task, mesos resource offer for slave node A will be sent to Jenkins Master B, then Jenkins Master B can start task.
The slave can be used by Jenkins Mesos A & Jenkins Mesos B, but never at the same time.
Hope it help you.
We have a Jenkins server with 8 executors and 20 jobs. 15 of those jobs take approximately 2 hours to finish while the remaining 5 take only 15 minutes. I would like to reserve 1 executor (or 2) to run those 5 small jobs only and restrict other jobs to run on the other executors. Note: I don't have any slaves, just 8 executors on master Jenkins process.
I'm new to Jenkins so I just wonder is it any way that I can do that? Thank you.
As i understand it Kiddo uses the master for 8 executors. What you can do is to add a new slave which runs on the master, let's call it slave-master. I.e. You will have master with 6 executors that has usage set to utilise as much as possible, and then slave-master which has usage restricted to only the short builds. So on your server you will have two jenkins tasks running, one is the jenkins master it self, and two is the slave-master.
For info on how to connect slaves, go to https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds
Adding to #StephenKing answer, you also have to specify the label name for each job while configuring it, as shown in the below image:
I'm a bit late but I think it would be much easier to restrict how many concurrent "slow" jobs can run than trying to reserve executors. This is simple to do with the Lockable Resources plugin: https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin
Simply add as many resources as the number of slow jobs you want to allow (6 or 7) and give them all the same label. Modify the job configurations to lock a resource (by label with quantity 1) before it can execute. If all the resources are already locked, then the job will wait until one is freed.
In the slave configuration, you can set the Usage mode to Only build jobs with label expressions matching this node.
Then, only jobs matching a given label (e.g. job-group-whatever) will be executed on this slave.
I had same issue. I installed multiple agent on same slave and it works fine.
Nodes remote directory should be different.
agent as a windows services
I have several integration tests within my Jenkins jobs. They run on several application servers, and I want to make sure that only one integration test job is run at the same time on one application server.
I would need something like a tag or variable within my jobs which create a group of jobs and then configure the logic that within that group, only one job may run at the same time.
Could I use the Exclusion plugin for that? Does anyone have experience with it?
Use the Throttle Concurrent Builds Plugin. It replaces the Locks and Latches plugin, and provides the capability to restrict the number of jobs running for specific labels.
For example: you create a project category 'Integration Test Server A' and tie jobs to it with a maximum concurrent count of 1, and a second 'Integration Test Server B' label and tie other jobs to it, both categories will only run a single concurrent build (assuming you've set a max job count of 1), and the other jobs in that category will queue until the 'lock' has cleared.
Using this method, you don't have to restrict the number of executors available on any specific Jenkins instance, and can easily add further slaves in the future without having to reconfigure all your jobs.
If I understand you right, you have a pool of application servers and it doesn't matter on what server your tests run. They only need to be the only test on that server.
I haven't seen a plugin that can do that. However, you can get easily around it. You need to configure a slave for each application server. (1 slave = 1 app server) You need to assign the same label to all slaves and every slave can only have one executor. Then you assign the jobs that run the integration tests, to run on that label. Jenkins will assign the jobs then to the next available slave (or node) that has that label.
Bare in mind that you can have more than one slave running on the same piece of hardware and even a master and a slave can coexist on the same server.
Did you check below parameter in the Jenkins -> Manage Jenkins -> Configure system
# of executors
The above parameter helps you restrict the number of jobs to be executed at a time.
A Jenkins executor is one of the basic building blocks which allow a build to run on a node/agent (e.g. build server). Think of an executor as a single “process ID”, or as the basic unit of resource that Jenkins executes on your machine to run a build. Please see Jenkins Terminology for more details regarding executors, nodes/agents, as well as other foundational pieces of Jenkins.
You can find information on how to set the number of Jenkins executors for a given agent on the Remoting Best Practices page, section Number of executors.
Source - https://support.cloudbees.com/hc/en-us/articles/216456477-What-is-a-Jenkins-Executor-and-how-can-I-best-utilize-my-executors