Let's say I have a job A and also a agent configured. I want to run build 1 of Job A on master and build 2 of Job A on agent node.
Is there an option to achieve that ?
OR
Is there a way where my job looks at controller and if it already finds a build running, then start the next build on agent ?
Are you intending to run in parallel or just alternate? (Not a good idea to run jobs on master; could configure a node to run on same host as "master".). Seems to be parallel and you have restricted to one executor each on master and agent (you can have more, in which case any advice may be moot).
Nevertheless, Jenkins queue job allocation to executors is "sticky"; it tries to run where last run, unless unavailable. This can lead to overloading in nodes. So the M,A,M,A pattern is unnatural.
There are plugins that might help: Least Load, Scoring Load Balancer, but maybe not.
Perhaps an approach would be to restrict your job using a label and have a post-build groovy step that moves the label to the other upon success for the next run or two labels and the job self-modifies the label to match the other.
Related
What I'd like to do:
Before/after any build allocates a slave: Reset slave to a pristine state (eg restore/delete files, kill rogue processes, etc. In other words, run some script). I know how to do this in one job, I don't know how to have Jenkins trigger it for every job of every type without modifying them all.
Unsatisfactory Ways I can think of accomplishing this:
Enforce a pre/post step in every AbstractProject (old style jobs). This is only scalable if using the Job DSL Plugin to generate all jobs.
Enforce a pre/post step in every PipelineJob. This is only scalable if using a Jenkins shared library (eg: myBuild() that encapsulates these steps).
Make use of the Global Post Script plugin and run a groovy script after every build. This works, but the script runs after the slave has been released, so another build may have grabbed it already (it's too late). Figuring out what nodes were allocated during the build is also fairly complicated
Switch all slaves to one-shot type (ie Docker) that require no clean-up. This doesn't work for my use cases, but may work for someone else.
Periodically run a job that uses a System Groovy Script to edit all other jobs to add a "clean-up" "pre" build step if not present. (Will not work for Pipeline jobs)
(I currently do #1 and #2)
Ideal Solution
Theoretical options:
A plugin to hook into some event (ie WorkspaceListener.beforeUse() ) and execute something then (unfortunately, WorkspaceListener does not apply to Pipeline Jobs). This should trigger right before a slave is used. It gets a little complicated when a slave has multiple executors (mine don't).
A plugin to enforce execution of some steps, similar to "Execute builders from job X", but in every job. (also doesn't work for Pipeline jobs)
Assuming that I'm using Swarm for all slaves, modify the swarm client to handle this logic (perform a task when slave becomes idle). A poor man's way would be to make the swarm client run in "one-shot" mode, in a bash loop.
Question:
What am I overlooking? Is there a better way?
Let’s say I have Jenkins job that should be running on a slave for several hours, is there a way that while job is running to run it but this time on a different slave so same job will run on 2 salves in parallel ?
Currently when I try to do that I get something like that:
(pending—Build # is already in progress )
You will need to check "Execute concurrent builds" in the job.
You will also have to install https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin
This will allow you to specify how many runs you want your job to execute concurrently and how many runs per slave - so you won't have to limit the executors on your slaves to one, and that way you can always assure there will be no 2 parallel runs on one slave.
Good luck!
I have Jenkins installed on my computer which configure to have just the master as a node (no other nodes) and number of executes is 5. I created a Job named "myJob" and I want to run it on the master 2 times at the simultaneously (meaning that if I run Builds 90 and 91, I don't want to get the message that "pending—Build #90 is already in progress").
I also have Throttle Concurrent Builds plugin installed and it allows this job to run more than once at the same time..
I still getting the "pending" message.
Can anyone tell me how to achieve it?
By default, multiple builds of the same job cannot run at the same time — that's why it remains in the queue with the "already in progress" message.
You can, however, change this behaviour in a job's configuration by enabling the "Execute concurrent builds if necessary" checkbox.
Note that, if you have multiple build nodes, the two concurrent builds of the job will likely be spread out to run on different nodes, rather than occupying both executors of the same build node.
If you require that they both run on the same machine, you should likely use the "Restrict where this job can run" option to ensure all builds of that job run on the same node.
You also need to make sure the salve is set for running more than one job.
To set your note to run more than one job:
Manage Jenkins -> Manage Nodes -> Configure Node -> # of executors
Has to set to 2 or whatever jobs you want to execute simultaneously.
As previously mentioned, you need to check the "Execute concurrent builds" box in the job's config screen.
You will then get a "job\workspace" and a "job1\workspace" (or something similar.) Since both jobs need their own workspace, the second instance will need to create a new workspace folder structure.
Just something to keep in mind if you expected both jobs to be running in the same directory.
I have Jenkins setup of 6 Slaves and master, all windows machines. Now I have a housekeeping Jenkins job which I want to periodically run on all the slaves and master, as this job does following tasks
Delete unused temporary files.
Delete unwanted processes, as some of the tests are leaking processes (why leak is different question).
Set certain environment variables, as sometimes I want to push environment variable changes to all machines.
Any idea how can I force Jenkins to run this one job on all slaves and master once every day? As a work around I can create multiple Jenkins job and mark each one to run on one particular slave or master, but I would rather avoid having so many duplicate jobs.
The Node and Label Parameter plugin allows you to parameterize where a job should be run. The job can be run on more than one node -- each node shows up as a separate execution in the job's build history. When multiple nodes are selected, you can configure whether the job should continue to run on other nodes if an execution fails.
I had a similar need, but using the Node and Label Parameter Plugin didn't seem quite right, as I do not want to parameterize my cleanup jobs.
I found a more satisfying answer in this post and thought it would also benefit to this question: Jenkins - Running a single job in master as well as slave.
Here is some documentation on how to configure a "Matrix project": https://wiki.jenkins.io/display/JENKINS/Building+a+matrix+project.
What you are looking for is the "Slave axis". It's not very well documented in the page above, but it appears as an option of the "Add axis" menu whenever there are more than one node. Here's a screenshot of the interesting part:
Updates according to recent Jenkins
Pipeline type:
On "Configuration" page for pipeline:
I have several integration tests within my Jenkins jobs. They run on several application servers, and I want to make sure that only one integration test job is run at the same time on one application server.
I would need something like a tag or variable within my jobs which create a group of jobs and then configure the logic that within that group, only one job may run at the same time.
Could I use the Exclusion plugin for that? Does anyone have experience with it?
Use the Throttle Concurrent Builds Plugin. It replaces the Locks and Latches plugin, and provides the capability to restrict the number of jobs running for specific labels.
For example: you create a project category 'Integration Test Server A' and tie jobs to it with a maximum concurrent count of 1, and a second 'Integration Test Server B' label and tie other jobs to it, both categories will only run a single concurrent build (assuming you've set a max job count of 1), and the other jobs in that category will queue until the 'lock' has cleared.
Using this method, you don't have to restrict the number of executors available on any specific Jenkins instance, and can easily add further slaves in the future without having to reconfigure all your jobs.
If I understand you right, you have a pool of application servers and it doesn't matter on what server your tests run. They only need to be the only test on that server.
I haven't seen a plugin that can do that. However, you can get easily around it. You need to configure a slave for each application server. (1 slave = 1 app server) You need to assign the same label to all slaves and every slave can only have one executor. Then you assign the jobs that run the integration tests, to run on that label. Jenkins will assign the jobs then to the next available slave (or node) that has that label.
Bare in mind that you can have more than one slave running on the same piece of hardware and even a master and a slave can coexist on the same server.
Did you check below parameter in the Jenkins -> Manage Jenkins -> Configure system
# of executors
The above parameter helps you restrict the number of jobs to be executed at a time.
A Jenkins executor is one of the basic building blocks which allow a build to run on a node/agent (e.g. build server). Think of an executor as a single “process ID”, or as the basic unit of resource that Jenkins executes on your machine to run a build. Please see Jenkins Terminology for more details regarding executors, nodes/agents, as well as other foundational pieces of Jenkins.
You can find information on how to set the number of Jenkins executors for a given agent on the Remoting Best Practices page, section Number of executors.
Source - https://support.cloudbees.com/hc/en-us/articles/216456477-What-is-a-Jenkins-Executor-and-how-can-I-best-utilize-my-executors