We use jenkins for automation for our test infrastructure. The requirement is to give users the ability to use a jenkins node for their private test or debug using private jenkins jobs and then put back in the pool of nodes marked with labels; so that other jobs that were marked to run on particular labels can be run without interference.
We can achieve this by letting users alter label, but that didnt workout as users (nearly 50) are making their own label names and it takes time for admin to reassign the nodes (even with process) and precious test time is getting affected.
we are looking for some solution such as ability to provide buttons like take this node offline (cant use this option since jenkins cannot see the node anymore and so users cannot run jenkins job on the node) but may be with the ability to run scripts.
I have done some research on this but have to compromise on some requirements, so i decided to seek help from the community... SUGGESTIONS?
Did you have a look to this question:
How to take Jenkins master node offline using CLI?
In the 1st question, there are some CLI to make a node offline.
Maybe you can create a dedicated job on the master with one parameter (the node name). This job will call the Jenkins CLI to stop your node.
Related
I have a Jenkins master and two agents. However the connectivity to one agent(agentA) is bit shaky and I want to use the other agent(agentB) when the connectivity to the first one is not available.
I am only using the Jenkins web interface and have not used scripts. I am trying to figure out how it can be done using the "Restrict where this project can be run" option in job's configuration. I tried using agentA|| agentB but when agentA is not available it hangs saying "pending - agentA is offline"
Is it possible to have a configuration to achieve what I need?
I can;t leave it blank because I have other agent (agentC, agentD) which do not want this job to run in.
I am not an admin of the Jenkins server, hence adding new plugins is not my preferred option but it can be done.
As noted in Least Load plugin,
By default Jenkins tries to allocate a jobs to the last node is was executed on. This can result in nodes being left idle while other nodes are overloaded.
As you generalized the example, I'm not 100% sure if your situation can be solved by simply better labelling of your nodes or you want to look at least load plugin (it is designed for balancing the load across nodes). Your example appears to show Node names (ie; agentA/agentB). The Queue allocation logic may be "Only A or Only B", then Jenkins sticks to it. Load balancing may not address that as while a Node (a Computer) name is also a label, it may have additional logic tied to it.
If you label the pair of nodes in a pool with a common label, say "CapabilityA", and constrain your jobs to run where "CapabilityA" rather than the node names, you may find jobs float across the pool (to B if A is not available. That's how we have our nodes labelled - by Capability, and we see jobs floating across nodes, but only once the first node is full (4 executors each), so not balanced.
Nodes can have many labels and you can use label conditions to have complex constraints.
The way we're using Jenkins requires us to have two nodes defined for each machine. One Jenkins node runs as a normal user (called Normal), and the other runs as the administrator (called Admin). So they show up as two separate nodes, even though they exist on the same slave machine.
But, we're running into a concurrency problem. Because our job switches between the two nodes, there is a possibility of another job (Job B) being assigned to (for example) the Normal node, while the Admin node is working on its part of (e.g.) Job A.
Is there a way to tell Jenkins that if either the Normal node or the Admin node of a machine is being used, then it should NOT give the other node some other job?
To elaborate on this question--we have a test suite that we currently run serially. All of our Jenkins masters have multiple slaves, so naturally we would like to take advantage of parallelization, so the suite doesn't spent 2 hours using one machine while the other ones sit idle. So it's not really a matter of ensuring only one job runs at once, it really is a matter of telling Jenkins not to use a node when its partner node is busy.
The issue is not related to two nodes on the same machine or one privileged or not; it's a matter of blocking one job from running while the other is still running.
I trust you are using labels to restrict what jobs can run on what nodes
You can use Build Blocker plugin to block the job from running while others are . There are other plugin options which may work for you as well.
You can also use the Paramterized Trigger to in-line the execution of the other job. It can be run as a build step or a post-build step.
You can also restrict the number of executors on a given node via ${JENKINS_URL}/computer/myNode/configure | # of executors, so you don't run multiple jobs one the same node if that's an issue.
Here's the way I solved this problem:
Set the number of executors on each slave node to 1.
Force my job to take an executor for the whole length of the job.
Specifically, in the groovy script that we use for all our jobs, at the very top, after we find which two (admin and normal, running on the same slave) nodes we need, we use the following:
n
node(myNormalNode)
{
//all the rest of the job, including:
node(myAdminNode)
{
//other commands for the admin node
}
//back to commands for the normal node
node('master')
{
//code to run on master
}
//and so forth
}
This makes Jenkins not assign any other jobs to this computer until the first job is done.
Got a pipeline job who can run at 4 different nodes with one label. Previously i got the problem that they randomly tried to run at the same node, so i installed the lockable recources plugin and tried this:
node('TEST') {
try {
notifyBuild('STARTED')
lock(env.NODE_NAME){
This works generally, but it seems to be random wich node from the Label TEST the job chooses. For example the first two job executions can choose the same node and so the 2nd job will have to wait even if there are free nodes available. Is there a way to secure that all nodes are used before jobs have to wait?
Better solution is the https://github.com/jenkinsci/throttle-concurrent-builds-plugin which also works for pipeline jobs. This plugin doesn´t checks if recources are available before it blocks them. Also all recources are used before jobs have to wait.
We need to be able to run a Jenkins job that consumes two slaves. (Or, two jobs, if we can guarantee that they run at the same time, and it's possible for at least one to know what the other is.) The situation is that we have a heavy weight application that we need to run tests against. The tests run on one machine, the application runs on another. It's not practical to have them on the same host.
Right now, we have a Jenkins job that uses a script to kick a dedicated application server up, install the correct version, the correct data, and then run the tests against it. That means that I can't use the dedicated application server to run other tasks, when there aren't the heavy weight testing going on. It also pretty much limits us to one loop. Being able to assign the app server dynamically would allow more of them.
There's clearly no way to do this in the core jenkins, but I'm hoping there's some plugin or hackery to make this possible. The current test build is a maven 2 job, but that's configurable, if we have to wrap it in something else. It's kicked off by the successful completion of another job, which could be changed to start two, or whatever else is required.
I just learned from that the simultaneous allocation of multiple slaves can be done nicely in a pipeline job, by nesting node clauses:
node('label1') {
node('label2') {
// your code here
[...]
}
}
See this question where Mateusz suggested that solution for a similar problem.
Let me see if I understood the problem.
You want to have dynamically choose a slave and start the App Server on it.
When the App server is running on a Slave you do not want it to run any other job.
But when the App server is not running, you want to use that Slave as any other Slave for other jobs.
One way out will be to label the Slaves. And use "Restrict where this project can be run" to make the App Server and the Test Suite run on the machines with Slave label.
Then in the slave nodes, put " # of executors" to 1. This will make sure that at anytime only one Job will run.
Next step will be to create a job to start the App Server and then kick off the Test job once the App Server start job is successful..
If your test job needs to know the server details of the machine where your App server is running then it becomes interesting.
Given several nodes available Jenkins nodes (NodeA, NodeB, ...), is there a way to define a job such that it gets 2 nodes?
For example, Job1 gets assigned to NodeA & NodeD.
To make a job run TWICE - both on node-A AND node-D, use the Matrix-build Plugin (note it is a little tricky to configure and monitor).
To make a job run ONCE - either on node-A OR node-D, give those two nodes the same LABEL, and RESTRICT the job to run on that Label.
Cheers
You'll need to submit the job twice and use the NodeLabel Parameter Plugin to choose which node to execute each job on. To submit both these jobs at the same time, you could try using the Build Flow plugin
Why are you trying to do this? Are you perhaps trying to use slave agents to install software? Perhaps you could consider using an orchestration tool like rundeck (Which also has a Jenkins plugin)