The way we're using Jenkins requires us to have two nodes defined for each machine. One Jenkins node runs as a normal user (called Normal), and the other runs as the administrator (called Admin). So they show up as two separate nodes, even though they exist on the same slave machine.
But, we're running into a concurrency problem. Because our job switches between the two nodes, there is a possibility of another job (Job B) being assigned to (for example) the Normal node, while the Admin node is working on its part of (e.g.) Job A.
Is there a way to tell Jenkins that if either the Normal node or the Admin node of a machine is being used, then it should NOT give the other node some other job?
To elaborate on this question--we have a test suite that we currently run serially. All of our Jenkins masters have multiple slaves, so naturally we would like to take advantage of parallelization, so the suite doesn't spent 2 hours using one machine while the other ones sit idle. So it's not really a matter of ensuring only one job runs at once, it really is a matter of telling Jenkins not to use a node when its partner node is busy.
The issue is not related to two nodes on the same machine or one privileged or not; it's a matter of blocking one job from running while the other is still running.
I trust you are using labels to restrict what jobs can run on what nodes
You can use Build Blocker plugin to block the job from running while others are . There are other plugin options which may work for you as well.
You can also use the Paramterized Trigger to in-line the execution of the other job. It can be run as a build step or a post-build step.
You can also restrict the number of executors on a given node via ${JENKINS_URL}/computer/myNode/configure | # of executors, so you don't run multiple jobs one the same node if that's an issue.
Here's the way I solved this problem:
Set the number of executors on each slave node to 1.
Force my job to take an executor for the whole length of the job.
Specifically, in the groovy script that we use for all our jobs, at the very top, after we find which two (admin and normal, running on the same slave) nodes we need, we use the following:
n
node(myNormalNode)
{
//all the rest of the job, including:
node(myAdminNode)
{
//other commands for the admin node
}
//back to commands for the normal node
node('master')
{
//code to run on master
}
//and so forth
}
This makes Jenkins not assign any other jobs to this computer until the first job is done.
Related
I am working on tests of embedded devices.
We have a device farm where several types of devices (say A, B and C), with many instances each (A1..A10, B1..B10, C1..C10), are available for remote testing. Each device is connected to a Jenkins node (with label 'A', 'B' or 'C'). So far, if we wanted to test systems of devices (A, B and C working together), they were connected to many Jenkins nodes ("multi-node setup") which had to be addressed individually.
Now we would like to replace these with single-node setups (all embedded devices A, B, C controlled by the same node). The current job structure involved reserving all nodes from a given multi-node setup on a high level and then calling different jobs one by one which then run on the respective node.
For reasons of maintainability and clear responsibility, I would like to retain the structure of several sub-jobs for given test.
My problem is now that the top-level job chooses any of the single-node setups, but of course all downstream sub-jobs need to run on that exact instance, which I can assure by passing the NODE_NAME of the top-level job as parameter. Unfortunately, since I should never have two (top-level) jobs running on a same instance in parallel, I can only define one executor per node. Which, in turn, means that a downstream job supposed to run on the same node as the top-level job will always wait for the executor to become available infinitely.
TL;DR: Is there a way to allow a downstream job to "reuse" an executor from an upstream job?
I am perfectly ok with "No, there isn't", but it would make managing our tests easier if there were. Otherwise I'll just have to roll all jobs into one big one.
Got a pipeline job who can run at 4 different nodes with one label. Previously i got the problem that they randomly tried to run at the same node, so i installed the lockable recources plugin and tried this:
node('TEST') {
try {
notifyBuild('STARTED')
lock(env.NODE_NAME){
This works generally, but it seems to be random wich node from the Label TEST the job chooses. For example the first two job executions can choose the same node and so the 2nd job will have to wait even if there are free nodes available. Is there a way to secure that all nodes are used before jobs have to wait?
Better solution is the https://github.com/jenkinsci/throttle-concurrent-builds-plugin which also works for pipeline jobs. This plugin doesn´t checks if recources are available before it blocks them. Also all recources are used before jobs have to wait.
I have a very lightweight job that should be executed immediately when it is triggered, rather than waiting an hours for current jobs to finish.
As I understand, a flyweight task is what I want. It will create a ephemeral executor, just for that task.
How can I make a job be run as flyweight?
I have recently had the same problem. My company has a lot of jenkins projects and some have more precedence over others, and we limit the number of executors to only 4.
Therefore, we decided to create some slaves, instead of always building on the master. Create a slave node that only builds your "very lightweight job".
Go to Manage Jenkin -> Manage nodes -> New node -> Dumb slave.
Then configure your slave node to your liking. Now configure the "very lightweight job". Make sure that This build is parameter is checked, then Add parameter -> Node.
Then select the slave node that you just created. There are a lot of configurations, such as which node do you want to default, but I think you can customize that to your liking.
Try out this FlyWeightProject Plugin. It is an extension of the Freestyle type that runs in Flyweight.
AFAIU the issue is that all your executors are occupied when it comes to running this high priority job.
What about:
Establishing another slave node; in a VM, for instance; with an appropriate number of executors, in case there are more of these high priority jobs
Assigning a label like high-priority to this new node
Restricting where this high priority job can be run to this high-priority label
Assigning a label like long-running to all other nodes
Restricting where all other jobs can be run to this long-running label
Another possibility is to:
Configure the new node's Usage: Only build jobs with label restrictions matching this node
Restrict where the high priority job can be run to the new node
This avoids having to create and assign labels to all jobs, as mentioned above, but is less flexible for future adaptions and extensions.
We use jenkins for automation for our test infrastructure. The requirement is to give users the ability to use a jenkins node for their private test or debug using private jenkins jobs and then put back in the pool of nodes marked with labels; so that other jobs that were marked to run on particular labels can be run without interference.
We can achieve this by letting users alter label, but that didnt workout as users (nearly 50) are making their own label names and it takes time for admin to reassign the nodes (even with process) and precious test time is getting affected.
we are looking for some solution such as ability to provide buttons like take this node offline (cant use this option since jenkins cannot see the node anymore and so users cannot run jenkins job on the node) but may be with the ability to run scripts.
I have done some research on this but have to compromise on some requirements, so i decided to seek help from the community... SUGGESTIONS?
Did you have a look to this question:
How to take Jenkins master node offline using CLI?
In the 1st question, there are some CLI to make a node offline.
Maybe you can create a dedicated job on the master with one parameter (the node name). This job will call the Jenkins CLI to stop your node.
Given several nodes available Jenkins nodes (NodeA, NodeB, ...), is there a way to define a job such that it gets 2 nodes?
For example, Job1 gets assigned to NodeA & NodeD.
To make a job run TWICE - both on node-A AND node-D, use the Matrix-build Plugin (note it is a little tricky to configure and monitor).
To make a job run ONCE - either on node-A OR node-D, give those two nodes the same LABEL, and RESTRICT the job to run on that Label.
Cheers
You'll need to submit the job twice and use the NodeLabel Parameter Plugin to choose which node to execute each job on. To submit both these jobs at the same time, you could try using the Build Flow plugin
Why are you trying to do this? Are you perhaps trying to use slave agents to install software? Perhaps you could consider using an orchestration tool like rundeck (Which also has a Jenkins plugin)