I have a Jenkins pipeline that has 6 stages.
In 4 of the 6 stages the pipeline triggers Jenkins jobs using build job.
Both the pipeline and the job are set to allow concurrent parallel executions.
The node on which the pipeline and the jobs run is a single node having number of executer set to 10.
Everything works fine when I run 10 parallel concurrent pipeline.
But if I run more than 10 parallel pipelines then all of them seem to go in a deadlock, none of them completes no matter how long you wait and send they are waiting on each other to complete.
If I kill the 11th execution then all the 10 start completing successfully.
My requirement is that if someone executes more concurrent builds of a pipeline than the number of executors it has for the node that it runs on; then in that case 10 should complete parallel execution and 11 onwards should wait until then and should be executed in the second batch of 10 executions rather than all going to a hung state.
Please help me understand if this is a bug with Jenkins latest version and what is the workaround to avoid all pipeline builds from falling in hung state?
The issue can be that the master or the node runs out of CPU and/or memory.
You can also look at Jenkins master/slave node logs for exceptions.
Related
I'm running a single-node Jenkins instance on a project where I want to clean up closed PRs intermittently. The system can keep up just fine with the average PR submission rate; however, when we re-run the scan repository against the project the system encounters a ton of parallel build congestion.
When we run scan repository the multi-branch pipeline fires off the separate stages and those generally complete sequentially without issue and without hitting timeouts; however, the priority of declarative post actions seems lower than the other stages (no special plugins are installed that would cause that AFAIK). The behavior exhibited is that parallel builds start running impacting the total run-time for any one branch or pull request such that all of the builds might indicate say 60 or 90 minute build times instead of the usual 6-10 minutes when the only task remaining is filling out the checkstyle reports or whatever minor notifications tasks there are.
Is there a way to dedicate a single executor thread on the master node for declarative post actions so one branch or PR can be ran from end-to-end without being stuck waiting for available executors that have suddenly been picked to start a different PR or branch and run the earlier (and computationally expensive stages like linting code and running unit tests) in order to avoid ultimately hitting a timeout?
Recently I see some strange behavior with Jenkins, configured to use 5 executors (on master) only, but despite the configuration, I see this:
My build queue has few hundreds of jobs waiting and dispatching correctly.
But I can't understand why there are so many running jobs on my master if only 5 executors configured?
Running Jenkins ver. 2.89.1
I guess you are using Pipeline jobs. All the pipeline code which is not inside a node is executed in a Jenkins master thread and it's shown in the UI as an executor, usually called lightweight executor. They do not consume a real executor in a node.
This is why your pipelines should not do any heavy work other than orchestrating the build. Any real build (calls to build tools, computation or resources demanding tasks) should be done inside a node block.
Trouble aborting pipeline jobs
We recently converted some of our jobs over to pipeline jobs (specifically, multibranch pipeline jobs), and since we did so, stopping builds has become much more problematic.
Back when they were freestyle jobs, the builds would immediately stop when we clicked the red X next to a build.
But when we do that in a pipeline job, often it won't stop. In the console output, we'll get something like this:
Aborted by [USERNAME]
Sending interrupt signal to process
Click here to forcibly terminate running steps
We have to click that link in the third line, and then it often leaves mercurial repositories in a bad state.
Why did stopping builds work fine with freestyle jobs, but not with Pipeline jobs, and is there any way to get them working well with Pipeline jobs?
I have Job A which triggers two downstream jobs, say Job B and Job C. These two downstream jobs executes in parallel on master node. Job B then triggers another job say job 1 on a slave node and job C triggers job 2 on the same slave node almost at the same time. Number of executors configured on this slave node is 1 as i don't want job 1 and job 2 to run concurrently. Job 2 gets queued since job 1 is already running on that slave node. But when job 1 finishes execution, job 2 runs into executor starvation and gets stuck.
Issue:- Job 2 is running into executor starvation and gets stuck.
Is there any way in which once job 1 finishes execution, job 2 starts running instead of getting stack. i have also tried Throttle concurrent builds plugin but still the issue persists.
Thanks in advance!
There are several plugins that allow resource management:
https://wiki.jenkins-ci.org/display/JENKINS/Lockable+Resources+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin
https://wiki.jenkins-ci.org/display/JENKINS/External+Resource+Dispatcher
Some block the job from executing, others have the job "wait" for the resource (while job is running).
You would have to setup your Job1 and Job2 to use some resource, so that they can wait for one another.
I have a Jenkins server with half a dozen builds. Each of these builds is composed of a parent job that triggers anywhere between 4 and 6 smaller jobs that run in parallel. I use the EC2 plugin to keep the number of active slaves in line with the number of queued builds. Or in other words, slaves are coming and going all the time. Each slave has 7 executors (parent job + max(4, 6)).
It is absolutely crucial that all jobs of a build are executed on the same machine. I also cannot allow any jobs from build A to execute on a machine that has jobs from build B running.
What I'm looking for is a way that prevents Jenkins from using any inactive executors of a node as long as any jobs from a previous build are still active on it.
I've spent the day experimenting with a combination of the Throttle Concurrent Builds Plugin and the NodeLabel Parameter Plugin. Unfortunately, there seems to be a bug somewhere that causes throttled builds to not contribute to the Load Statistics of a slave. This in turn means that throttled build will never trigger Jenkins to spin up additional slaves. Obviously this is totally unacceptable.
You can try and use "This build is parameterized"
and pass the $NODE_NAME as a parameter between the builds and then use it at the "Restrict where this project can be run"