Jenkins won't execute any jobs. Having viewed this question, I have disabled all slave nodes but a simple job won't even run on the Master node.
What is wrong?
The Jenkins admin console can run, even with the Master node offline. This can happen when Jenkins runs out of disk space.
To confirm, do the following (with thanks to geekride - jenkins-pending-waiting-for-next-available-executor):
go to Jenkins -> Manage Jenkins -> Manage Nodes
examine the "master" node to see if it is offline. It may be reporting that the master node is out of disk space.
go to Jenkins -> Manage Jenkins -> Manage Nodes
examine the "master"
node.(Click on configure icon)
In my case No of executors was set to 0.
Increased it and issue got fixed.
In my case, I had the following set in my JenkinsFile
node('node'){
...
}
There was no node called 'node', only master (the value had been left in there after following some basic tutorials). Changing the value to 'master' got the build working.
In my case it was caused by number of executors (I had 1) and running Jenkins Job (Project) from Pipeline (my pipeline script started other Job in Jenkins). It caused deadlock - my pipeline held executor and was waiting for its job, but the job was waiting for free executor.
The solution may be increasing of # of executors in Jenkins -> Manage Jenkins -> Manage Nodes -> Configure (icon on required node).
I ran into a similar problem because my master was set to "Leave this machine for tied jobs only." So, even though I disabled the Slave, Jenkins kept on bypassing the Master, looking for something else.
Go to Jenkins --> Manage Jenkins --> Manage Nodes, and click on the configure button of your master node (looks like a screwdriver and a wrench). Check the Usage and make sure it's on "Utilize this slave as much as possible."
In my case, I had just installed the "Authorize Project" plugin and incorrectly setup the strategy in "Manage Jenkins -> Configure Global Security -> Access Control for Builds" as "Run as anonymous". So 'anonymous' had no rights to execute the job.
Setting the first strategy as "Run as User who Triggered Build" unlocked the queued jobs.
I'm a little late to the game, but this may help others.
In my case my jenkins master has a shared external resource, which is allocated to jenkins jobs by the external-resource-dispatcher-plugin. Due to bug JENKINS-19439 in the plugin (which is in beta), I found that my resource had been locked by a previous job, but wasn't unlocked when that previous job was cancelled.
To find out if a resource is currently in the locked state, navigate to the affected jenkins node, Jenkins -> Manage Jenkins -> Manage Nodes -> master
You should see the current state of any external resources. If any are unexpectedly locked this may be the reason why jobs are waiting for an executor.
I couldn't find any details of how to manually resolve this issue.
Restarting jenkins didn't resolve the problem.
In the end I went with the brutal approach:
Remove the external resource
(see Jenkins -> Manage Jenkins -> Manage Nodes -> master -> configure)
Restart jenkins
Re-create the external resource
In my case, I noticed this behavior when the box was out of memory (RAM)
I went to Jenkins -> Manage Jenkins -> Manage Nodes and found an out of memory exception.
I just freed up some memory on the machine and the jobs started to go into the executors.
Short answer:
Kill all the jobs which are running on the master.
In my case there were 3 jobs hung on the master for more than 10 days which were unnoticed. We usually do not run any jobs directly on the master, everything is run on slaves. I killed these 3 jobs which were hung, automatically the executors on the slave started picking up jobs.
Point to note that even though we have 8 slaves only 1 slave was in this affected state.
[EDIT] We found the answer to why only one slave was in this affected state.
When a Jenkins slave goes down, all the pending jobs automatically get transferred over to the master. All the 3 hung jobs which I killed were from this slave, so its likely a connection issue between the master and this particular slave.
For me below solution worked.
Jenkins --> Manage Jenkins --> Manage Nodes --> master -> configure -->
Node properties --> Restrict Jobs execution at node - is enabled and given access to specific users. I have given access myself and then job started to run.
If Restrict Jobs execution at node is enabled scheduled tasks cannot run.
In my case it is similar to #Michael Easter: I got a problem in a job due to lack of disk space. I cleared some space, restarted Jenkins but still the problem persisted.
The solution was to go to Jenkins -> Manage Jenkins -> Manage Nodes and just Click on the button to update the status.
In my case I've to set Execute concurrent builds if necessary in job's General settings.
I had just added a stage with a docker agent. Since I only had one node, master, I had to tell the container to reuse the node from the earlier stages:
agent {
docker {
image 'bitnami/mongodb:latest'
reuseNode true
}
}
You can vote to have this be the default behavior for agents and prevent this kind of lock.
In case you have installed the Parameterized Trigger plugin, a job waiting in the build queue might be a known issue JENKINS-47792.
A workaround fix is to downgrade the Parameterized Trigger plugin to version 2.35.1.
Note that you may be required to downgrade dependencies, such as the git plugin, as well.
What worked for me: I finally noticed the Build Executor Status window on the left on the main Jenkins dashboard. I run a dev/test instance on my local system with 2 executors. Both were currently occupied with builds that were not running. Upon cancelling these to jobs, my third (pending) job was able to run.
For me I have to restart the executors manually. Click on "Dead" under "Build Executor Status" and push the restart button.
I ran into a similar problem because my master was set to " # of executor (The maximum number of concurrent builds that Jenkins may perform on this agent).
Go to Jenkins --> Manage Jenkins --> Manage Nodes, and click on the configure button of your master node (increase the number of executor to run mutiple jobs at a time).
You may have milestones set up and your job won't run until the previous job is complete.
In my environment (images jenkins:lts-jdk11, jnlp-agent-maven e jnlp-agent-maven:jdk11), doing a pull for update jenkins image, reports no connectivity problem on agents. But all jobs stayed blocked with message:
pending—Waiting for next available executor on
So, the solution for me was stop and make a pull in both agents image:
docker pull jenkins/jnlp-agent-maven (jdk8)
docker pull jenkins/jnlp-agent-maven:jdk11
when you met this
just restart jenkins mater ,then recovery.
Related
I have setup a Jenkins server on a machine (master node) and have setup one slave node as well.
I have added same label 'test' on both master node as well as slave node. The same label 'test' has also been added to 'Restrict where this project can be run' property of my jenkins project.
The issue is, when I trigger 2 or more builds of this project, all are getting queued to run with one of the nodes only. (even when the other node is idle)
My expectation is if first build is running on slave node, second should run on the master node and vice-versa.
Anything I am missing here ?
NOTE - Tried configuring both the nodes with Usage property values 'Use this node as much as possible' as well as 'Only build jobs matching with label expression' but found the same behavior.
Thanks in Advance..!!
This issue is resolved now.
The 'Execute concurrent builds if necessary' option was NOT checked in my project because of which the executor was waiting for one build to complete before starting other build.
After checking this option, when 2 builds are triggered, they run simultaneously on master as well as slave node.
My job failed due to some reason. I want to go to the machine who actually run the job.
How do I know if a job is configured to use slaves?
How do I know which slave was used in that job?
By default all jobs are able to use all nodes, including the master as an executor. If you want to lock a job down to a particular slave, you can do so by selecting it in Job > Configuration > General > Restrict where this job can run.
To see which host a job ran on, click on the build number and in the top right it will say 'Started ~duration~ on ~host~', where ~host~ is your slave (or master).
Any plugins or default behavior in Jenkins that I could configure to force a build to be done on a different node if the previous build on a node was a failure?
To answer my own question, it appears there are not many plugins that allow you control over jobs running on Jenkins slaves and none that do this.
I have Jenkins setup of 6 Slaves and master, all windows machines. Now I have a housekeeping Jenkins job which I want to periodically run on all the slaves and master, as this job does following tasks
Delete unused temporary files.
Delete unwanted processes, as some of the tests are leaking processes (why leak is different question).
Set certain environment variables, as sometimes I want to push environment variable changes to all machines.
Any idea how can I force Jenkins to run this one job on all slaves and master once every day? As a work around I can create multiple Jenkins job and mark each one to run on one particular slave or master, but I would rather avoid having so many duplicate jobs.
The Node and Label Parameter plugin allows you to parameterize where a job should be run. The job can be run on more than one node -- each node shows up as a separate execution in the job's build history. When multiple nodes are selected, you can configure whether the job should continue to run on other nodes if an execution fails.
I had a similar need, but using the Node and Label Parameter Plugin didn't seem quite right, as I do not want to parameterize my cleanup jobs.
I found a more satisfying answer in this post and thought it would also benefit to this question: Jenkins - Running a single job in master as well as slave.
Here is some documentation on how to configure a "Matrix project": https://wiki.jenkins.io/display/JENKINS/Building+a+matrix+project.
What you are looking for is the "Slave axis". It's not very well documented in the page above, but it appears as an option of the "Add axis" menu whenever there are more than one node. Here's a screenshot of the interesting part:
Updates according to recent Jenkins
Pipeline type:
On "Configuration" page for pipeline:
My Jenkins job is a Matrix build that should run on build machines labeled AAA and BBB.
I have three build machines set up, each having label AAA and BBB.
However, when I start the build job, the job does not execute. Instead, it goes to "pending - Waiting for next available executor" state. Why does not my job execute?
Check the slave node configuration.
"Usage" field should be "Utilize this slave as much as possible" instead of "Leave this machine for tied jobs only".
Go to Manage Jenkins -> Configure System and increase the number of executor from 0 to 1
Go to Nodes > Configure > ## of executors. Increase the number here.
Try using Elastic Axis plugin. Afer installing it, in the multi configuration job you can find new axis added as Elastic axis. You just need to provide the label of the node. The job is built on all the nodes with that label. There is a check box provided to skip the nodes that are offline.
For me, already 2 jobs were in progress when I tried to execute the third one and hence I got this "Jenkins job is waiting for next available executor" on executing 3rd job.
The first two jobs were automatically triggered (as per my scripts) hence, I didn't realise they were running. After aborting these two jobs to run the third one, this error got resolved.
So, if you face this issue, just check once if any other job is already running. If yes, aborting that job or running after that job is completed may help resolve this issue.
I uninstalled Jenkins ,deleted all .jenkins files.Then I reinstalled jenkins, created job and build it successfully.