I have master jenkins and slave jenkins. I hav kept slave jenkins no of build executors as 1. Slave Jenkins also has 1 pipeline (Lets say pipeline A).
lets suppose a job from slave jenkins' own pipeline is running right now (Job A). I schedule a job from the master jenkins for slave jenkins (Job B).
I dont want Job B to run while job A is running as both jobs use shared resources.
Right now, Job B runs in parallel with Job A, which is causing Job A to fail.
How to do that?
Thanks!
Your implementation is a bit tricky since you are talking about 2 separate machines with 2 separate Jenkins instances. One option is to get rid of the Jenkins instance in the slave machine and move the Jenkins job that runs on it to the master machine. Then, you can schedule the job to use the resources of the slave machine while being managed by the master machine. If you do that, no further configuration will be needed since you have set the number of executors to 1.
If that is not possible, the other option is to find a way for them to communicate with each other that a build is running. Consider the third point of this answer. You can have a variable in a database somewhere and when one job starts, it updates the variable. Before the second job starts, it has to poll the variable to see if there is a job already running. If yes, the build doesn't start, if no, build starts and updates the variable.
Another less elegant solution is to simply have a text file in a location accessible to both machines and write the variable data into that instead of a database.
One way to do this is by using the Lockable Resources Plugin.
Let's say I have a job A and also a agent configured. I want to run build 1 of Job A on master and build 2 of Job A on agent node.
Is there an option to achieve that ?
OR
Is there a way where my job looks at controller and if it already finds a build running, then start the next build on agent ?
Are you intending to run in parallel or just alternate? (Not a good idea to run jobs on master; could configure a node to run on same host as "master".). Seems to be parallel and you have restricted to one executor each on master and agent (you can have more, in which case any advice may be moot).
Nevertheless, Jenkins queue job allocation to executors is "sticky"; it tries to run where last run, unless unavailable. This can lead to overloading in nodes. So the M,A,M,A pattern is unnatural.
There are plugins that might help: Least Load, Scoring Load Balancer, but maybe not.
Perhaps an approach would be to restrict your job using a label and have a post-build groovy step that moves the label to the other upon success for the next run or two labels and the job self-modifies the label to match the other.
I have Jenkins installed on my computer which configure to have just the master as a node (no other nodes) and number of executes is 5. I created a Job named "myJob" and I want to run it on the master 2 times at the simultaneously (meaning that if I run Builds 90 and 91, I don't want to get the message that "pending—Build #90 is already in progress").
I also have Throttle Concurrent Builds plugin installed and it allows this job to run more than once at the same time..
I still getting the "pending" message.
Can anyone tell me how to achieve it?
By default, multiple builds of the same job cannot run at the same time — that's why it remains in the queue with the "already in progress" message.
You can, however, change this behaviour in a job's configuration by enabling the "Execute concurrent builds if necessary" checkbox.
Note that, if you have multiple build nodes, the two concurrent builds of the job will likely be spread out to run on different nodes, rather than occupying both executors of the same build node.
If you require that they both run on the same machine, you should likely use the "Restrict where this job can run" option to ensure all builds of that job run on the same node.
You also need to make sure the salve is set for running more than one job.
To set your note to run more than one job:
Manage Jenkins -> Manage Nodes -> Configure Node -> # of executors
Has to set to 2 or whatever jobs you want to execute simultaneously.
As previously mentioned, you need to check the "Execute concurrent builds" box in the job's config screen.
You will then get a "job\workspace" and a "job1\workspace" (or something similar.) Since both jobs need their own workspace, the second instance will need to create a new workspace folder structure.
Just something to keep in mind if you expected both jobs to be running in the same directory.
How do we make Jenkins start the Job in a different slave every time it is run. It usually picks the slave which is free, but what if we wish to run the jobs in a different slave environment.
When you configure your job you should tick the box that says "Restrict where this project can be run" and add the name of the specific node you want your job to run on.
Jenkins won't execute any jobs. Having viewed this question, I have disabled all slave nodes but a simple job won't even run on the Master node.
What is wrong?
The Jenkins admin console can run, even with the Master node offline. This can happen when Jenkins runs out of disk space.
To confirm, do the following (with thanks to geekride - jenkins-pending-waiting-for-next-available-executor):
go to Jenkins -> Manage Jenkins -> Manage Nodes
examine the "master" node to see if it is offline. It may be reporting that the master node is out of disk space.
go to Jenkins -> Manage Jenkins -> Manage Nodes
examine the "master"
node.(Click on configure icon)
In my case No of executors was set to 0.
Increased it and issue got fixed.
In my case, I had the following set in my JenkinsFile
node('node'){
...
}
There was no node called 'node', only master (the value had been left in there after following some basic tutorials). Changing the value to 'master' got the build working.
In my case it was caused by number of executors (I had 1) and running Jenkins Job (Project) from Pipeline (my pipeline script started other Job in Jenkins). It caused deadlock - my pipeline held executor and was waiting for its job, but the job was waiting for free executor.
The solution may be increasing of # of executors in Jenkins -> Manage Jenkins -> Manage Nodes -> Configure (icon on required node).
I ran into a similar problem because my master was set to "Leave this machine for tied jobs only." So, even though I disabled the Slave, Jenkins kept on bypassing the Master, looking for something else.
Go to Jenkins --> Manage Jenkins --> Manage Nodes, and click on the configure button of your master node (looks like a screwdriver and a wrench). Check the Usage and make sure it's on "Utilize this slave as much as possible."
In my case, I had just installed the "Authorize Project" plugin and incorrectly setup the strategy in "Manage Jenkins -> Configure Global Security -> Access Control for Builds" as "Run as anonymous". So 'anonymous' had no rights to execute the job.
Setting the first strategy as "Run as User who Triggered Build" unlocked the queued jobs.
I'm a little late to the game, but this may help others.
In my case my jenkins master has a shared external resource, which is allocated to jenkins jobs by the external-resource-dispatcher-plugin. Due to bug JENKINS-19439 in the plugin (which is in beta), I found that my resource had been locked by a previous job, but wasn't unlocked when that previous job was cancelled.
To find out if a resource is currently in the locked state, navigate to the affected jenkins node, Jenkins -> Manage Jenkins -> Manage Nodes -> master
You should see the current state of any external resources. If any are unexpectedly locked this may be the reason why jobs are waiting for an executor.
I couldn't find any details of how to manually resolve this issue.
Restarting jenkins didn't resolve the problem.
In the end I went with the brutal approach:
Remove the external resource
(see Jenkins -> Manage Jenkins -> Manage Nodes -> master -> configure)
Restart jenkins
Re-create the external resource
In my case, I noticed this behavior when the box was out of memory (RAM)
I went to Jenkins -> Manage Jenkins -> Manage Nodes and found an out of memory exception.
I just freed up some memory on the machine and the jobs started to go into the executors.
Short answer:
Kill all the jobs which are running on the master.
In my case there were 3 jobs hung on the master for more than 10 days which were unnoticed. We usually do not run any jobs directly on the master, everything is run on slaves. I killed these 3 jobs which were hung, automatically the executors on the slave started picking up jobs.
Point to note that even though we have 8 slaves only 1 slave was in this affected state.
[EDIT] We found the answer to why only one slave was in this affected state.
When a Jenkins slave goes down, all the pending jobs automatically get transferred over to the master. All the 3 hung jobs which I killed were from this slave, so its likely a connection issue between the master and this particular slave.
For me below solution worked.
Jenkins --> Manage Jenkins --> Manage Nodes --> master -> configure -->
Node properties --> Restrict Jobs execution at node - is enabled and given access to specific users. I have given access myself and then job started to run.
If Restrict Jobs execution at node is enabled scheduled tasks cannot run.
In my case it is similar to #Michael Easter: I got a problem in a job due to lack of disk space. I cleared some space, restarted Jenkins but still the problem persisted.
The solution was to go to Jenkins -> Manage Jenkins -> Manage Nodes and just Click on the button to update the status.
In my case I've to set Execute concurrent builds if necessary in job's General settings.
I had just added a stage with a docker agent. Since I only had one node, master, I had to tell the container to reuse the node from the earlier stages:
agent {
docker {
image 'bitnami/mongodb:latest'
reuseNode true
}
}
You can vote to have this be the default behavior for agents and prevent this kind of lock.
In case you have installed the Parameterized Trigger plugin, a job waiting in the build queue might be a known issue JENKINS-47792.
A workaround fix is to downgrade the Parameterized Trigger plugin to version 2.35.1.
Note that you may be required to downgrade dependencies, such as the git plugin, as well.
What worked for me: I finally noticed the Build Executor Status window on the left on the main Jenkins dashboard. I run a dev/test instance on my local system with 2 executors. Both were currently occupied with builds that were not running. Upon cancelling these to jobs, my third (pending) job was able to run.
For me I have to restart the executors manually. Click on "Dead" under "Build Executor Status" and push the restart button.
I ran into a similar problem because my master was set to " # of executor (The maximum number of concurrent builds that Jenkins may perform on this agent).
Go to Jenkins --> Manage Jenkins --> Manage Nodes, and click on the configure button of your master node (increase the number of executor to run mutiple jobs at a time).
You may have milestones set up and your job won't run until the previous job is complete.
In my environment (images jenkins:lts-jdk11, jnlp-agent-maven e jnlp-agent-maven:jdk11), doing a pull for update jenkins image, reports no connectivity problem on agents. But all jobs stayed blocked with message:
pending—Waiting for next available executor on
So, the solution for me was stop and make a pull in both agents image:
docker pull jenkins/jnlp-agent-maven (jdk8)
docker pull jenkins/jnlp-agent-maven:jdk11
when you met this
just restart jenkins mater ,then recovery.