I have a scenario where I have N numbers of nodes and N number of tests, and these tests will be distributed to the nodes. My nodes have a label Windows.
Here's an example:
I have a pipeline job that will manage the distribution of the tests to the nodes. I set my pipeline job to run 10 tests on 10 VMs having the label Windows and it will run smoothly. However, one of my requirements is to concurrently run that pipeline job. The problem I might encounter If I have 10 tests on VMs 1-10 in the first run of the job, and run another job for 5 tests for VMs 11-15, given that I am using the Windows label, there might be a possibility that Jenkins will assign the test to VMs 1-10 but should run on VMs 11-15 or vice versa.
The solution I came up with is to dynamically change the label of the VMs from one of the jobs to a unique label that will only be used for that Job. Unfortunately, I still don't know how to do that.
Basically, I just want to logically group my nodes via label on demand in my pipeline script.
I searched all throughout the internet and yet I still wasn't able to find a solution that fits my needs.
Kindly help me with this. Also, I am open to using a different approach.
Related
I have a Jenkins master and two agents. However the connectivity to one agent(agentA) is bit shaky and I want to use the other agent(agentB) when the connectivity to the first one is not available.
I am only using the Jenkins web interface and have not used scripts. I am trying to figure out how it can be done using the "Restrict where this project can be run" option in job's configuration. I tried using agentA|| agentB but when agentA is not available it hangs saying "pending - agentA is offline"
Is it possible to have a configuration to achieve what I need?
I can;t leave it blank because I have other agent (agentC, agentD) which do not want this job to run in.
I am not an admin of the Jenkins server, hence adding new plugins is not my preferred option but it can be done.
As noted in Least Load plugin,
By default Jenkins tries to allocate a jobs to the last node is was executed on. This can result in nodes being left idle while other nodes are overloaded.
As you generalized the example, I'm not 100% sure if your situation can be solved by simply better labelling of your nodes or you want to look at least load plugin (it is designed for balancing the load across nodes). Your example appears to show Node names (ie; agentA/agentB). The Queue allocation logic may be "Only A or Only B", then Jenkins sticks to it. Load balancing may not address that as while a Node (a Computer) name is also a label, it may have additional logic tied to it.
If you label the pair of nodes in a pool with a common label, say "CapabilityA", and constrain your jobs to run where "CapabilityA" rather than the node names, you may find jobs float across the pool (to B if A is not available. That's how we have our nodes labelled - by Capability, and we see jobs floating across nodes, but only once the first node is full (4 executors each), so not balanced.
Nodes can have many labels and you can use label conditions to have complex constraints.
The way we're using Jenkins requires us to have two nodes defined for each machine. One Jenkins node runs as a normal user (called Normal), and the other runs as the administrator (called Admin). So they show up as two separate nodes, even though they exist on the same slave machine.
But, we're running into a concurrency problem. Because our job switches between the two nodes, there is a possibility of another job (Job B) being assigned to (for example) the Normal node, while the Admin node is working on its part of (e.g.) Job A.
Is there a way to tell Jenkins that if either the Normal node or the Admin node of a machine is being used, then it should NOT give the other node some other job?
To elaborate on this question--we have a test suite that we currently run serially. All of our Jenkins masters have multiple slaves, so naturally we would like to take advantage of parallelization, so the suite doesn't spent 2 hours using one machine while the other ones sit idle. So it's not really a matter of ensuring only one job runs at once, it really is a matter of telling Jenkins not to use a node when its partner node is busy.
The issue is not related to two nodes on the same machine or one privileged or not; it's a matter of blocking one job from running while the other is still running.
I trust you are using labels to restrict what jobs can run on what nodes
You can use Build Blocker plugin to block the job from running while others are . There are other plugin options which may work for you as well.
You can also use the Paramterized Trigger to in-line the execution of the other job. It can be run as a build step or a post-build step.
You can also restrict the number of executors on a given node via ${JENKINS_URL}/computer/myNode/configure | # of executors, so you don't run multiple jobs one the same node if that's an issue.
Here's the way I solved this problem:
Set the number of executors on each slave node to 1.
Force my job to take an executor for the whole length of the job.
Specifically, in the groovy script that we use for all our jobs, at the very top, after we find which two (admin and normal, running on the same slave) nodes we need, we use the following:
n
node(myNormalNode)
{
//all the rest of the job, including:
node(myAdminNode)
{
//other commands for the admin node
}
//back to commands for the normal node
node('master')
{
//code to run on master
}
//and so forth
}
This makes Jenkins not assign any other jobs to this computer until the first job is done.
Got a pipeline job who can run at 4 different nodes with one label. Previously i got the problem that they randomly tried to run at the same node, so i installed the lockable recources plugin and tried this:
node('TEST') {
try {
notifyBuild('STARTED')
lock(env.NODE_NAME){
This works generally, but it seems to be random wich node from the Label TEST the job chooses. For example the first two job executions can choose the same node and so the 2nd job will have to wait even if there are free nodes available. Is there a way to secure that all nodes are used before jobs have to wait?
Better solution is the https://github.com/jenkinsci/throttle-concurrent-builds-plugin which also works for pipeline jobs. This plugin doesn´t checks if recources are available before it blocks them. Also all recources are used before jobs have to wait.
trying to solve some problem with Mesos. I have three build servers for Jenkins. Jenkins schedules jobs on them through Mesos.
For now, Mesos loads one agent(slave) as hard as possible, but I want it to spread jobs across all agents..
As I see, it's better to run three jobs on three agents, than on one.
Is it possible to randomise job scheduling?
Or perhaps, I have such scenario. 2 large servers and one mini. I want to schedule Jobs on mini by default, and if it's not enough resources, then proceed to large servers. How can I achieve this goal? Is it possible to set priority for agents(slaves) to specify on which agent I want job to run at first?
The Mesos plugin for Jenkins attempts to build on the most recently built slave (see this method). This means that once it builds on that machine once, as long as that machine still has available spare resources - it'll schedule additional jobs on that machine until it is full. Right now it looks like that isn't optional (I have filed it as a feature request).
Given several nodes available Jenkins nodes (NodeA, NodeB, ...), is there a way to define a job such that it gets 2 nodes?
For example, Job1 gets assigned to NodeA & NodeD.
To make a job run TWICE - both on node-A AND node-D, use the Matrix-build Plugin (note it is a little tricky to configure and monitor).
To make a job run ONCE - either on node-A OR node-D, give those two nodes the same LABEL, and RESTRICT the job to run on that Label.
Cheers
You'll need to submit the job twice and use the NodeLabel Parameter Plugin to choose which node to execute each job on. To submit both these jobs at the same time, you could try using the Build Flow plugin
Why are you trying to do this? Are you perhaps trying to use slave agents to install software? Perhaps you could consider using an orchestration tool like rundeck (Which also has a Jenkins plugin)