How many Remote Nodes can Jenkins manage ? Are there any limitations/memory issues?
What is more effective:
1) 100 Nodes 1 executor per node ?
2) 5 Nodes with 20 executors per node ?
Tx.
As far as i know, there is no limitation on # of nodes one can have although your system might feel like saying, enough is enough! Issues such as number of processes per user (we got this issue recently, not with Jenkins but some other application where RAM and disk space were fine but the system stopped responding. We started getting system cannot fork() error), total number of open files etc. Few such issues might still be configurable but may not be allowed/feasible.
If resource (in your case, nodes) is not a constraint, which process wouldn't like to run wild? :) In practical cases, generally you wouldn't have the flexibility to opt for first option. In second case where you have 5 nodes with 20 executors, all you have to make sure is not to tie up jobs to a particular node unless you have a compelling reason.
Some slaves are faster, while others are slow. Some slaves are closer (network wise) to a master, others are far away. So doing a good build distribution is a challenge. Currently, Jenkins employs the following strategy:
If a project is configured to stick to one computer, that's always honored.
Jenkins tries to build a project on the same computer that it was previously built.
Jenkins tries to move long builds to slaves, because the amount of network interaction between a master and a slave tends to be logarithmic to the duration of a build (IOW, even if project A takes twice as long to build as project B, it won't require double network transfer.) So this strategy reduces the network overhead.
You should also have a look at these links:
https://wiki.jenkins-ci.org/display/JENKINS/Least+Load+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
Related
I have a Jenkins master and two agents. However the connectivity to one agent(agentA) is bit shaky and I want to use the other agent(agentB) when the connectivity to the first one is not available.
I am only using the Jenkins web interface and have not used scripts. I am trying to figure out how it can be done using the "Restrict where this project can be run" option in job's configuration. I tried using agentA|| agentB but when agentA is not available it hangs saying "pending - agentA is offline"
Is it possible to have a configuration to achieve what I need?
I can;t leave it blank because I have other agent (agentC, agentD) which do not want this job to run in.
I am not an admin of the Jenkins server, hence adding new plugins is not my preferred option but it can be done.
As noted in Least Load plugin,
By default Jenkins tries to allocate a jobs to the last node is was executed on. This can result in nodes being left idle while other nodes are overloaded.
As you generalized the example, I'm not 100% sure if your situation can be solved by simply better labelling of your nodes or you want to look at least load plugin (it is designed for balancing the load across nodes). Your example appears to show Node names (ie; agentA/agentB). The Queue allocation logic may be "Only A or Only B", then Jenkins sticks to it. Load balancing may not address that as while a Node (a Computer) name is also a label, it may have additional logic tied to it.
If you label the pair of nodes in a pool with a common label, say "CapabilityA", and constrain your jobs to run where "CapabilityA" rather than the node names, you may find jobs float across the pool (to B if A is not available. That's how we have our nodes labelled - by Capability, and we see jobs floating across nodes, but only once the first node is full (4 executors each), so not balanced.
Nodes can have many labels and you can use label conditions to have complex constraints.
I am new to Jenkins and I have configured master-slave nodes as shown below, but I need help to configure the no of executors in each of the below slave nodes
Currently, I have configured 100 executors in each slave nodes
How many no of executors I can configure in each slave node and what fact(memory, RAM, etc) need to take consider when increasing the no of executors?
Maximum how many no of executors I can configure in each server?
Well it totally depends on your usage. There are multiple factors such as how much cpu and memory is available, how the build are going to execute and what kind of builds, how frequent these build should run etc. But I can clearly say that 100 is too big number. I would suggest go with 20 builds (if builds run frequently and have fair amount of CPU and memory) first and observe if is there any issue with numbers or not then you can increase accordingly.
here is very nice article check this out https://www.avantica.com/blog/jenkins-balance-load-master-slave-setup#:~:text=Jobs%20are%20built%20using%20executors,to%20build%20two%20different%20tasks.
For a while, our Jenkins experiences critical problems. We have jobs hung, our job scheduler does not trigger the builds. After the Jenkins service restart, everything is back to normal, but after some period of time all problem are return. (this period can be week or day or ever less). Any idea where we can start looking? I'll appreciate any help on this issue
Muatik has made a good point in his comment, the recommended approach is to run jobs on agents (slave) nodes. If you already do it, you can look at:
Jenkins master machine CPU, RAM and hard disk usage. Access the machine and/or use plugin like Java Melody. I have seen missing graphics in the builds test results and stuck builds due to no hard disk space. You could also have hit the limit of RAM or CPU for the slaves/jobs you are executing. You may need more heap space.
Look at Jenkins log files, start with severe exceptions. If the files are too big or you see logrotate exceptions, you can change the logging levels, so that fewer exceptions are logged. For more details see my article on this topic. Try to fix exceptions that you see logged.
Go through recently made changes that can be the cause of such behavior, for example, new plugins, changes to config files (jenkins.xml)?
Look at TCP connections. Run netstat -a Are there suspicious connections (CLOSED_WAIT status)?
Delete old builds that you do not need.
We have been facing this issue from last 4 months, and tried everything, changing resources CPU & memory, increasing desired nodes in ASG. But nothing seems worked .
Solution: 1. Go to Manage Jnekins-> System Configurationd-> Maven project
configurations
2. In "usage" field, select "Only buid Jobs with label expressions matching this nodes"
Doing this resolved it and jenkins is working as a Rocket now :)
trying to solve some problem with Mesos. I have three build servers for Jenkins. Jenkins schedules jobs on them through Mesos.
For now, Mesos loads one agent(slave) as hard as possible, but I want it to spread jobs across all agents..
As I see, it's better to run three jobs on three agents, than on one.
Is it possible to randomise job scheduling?
Or perhaps, I have such scenario. 2 large servers and one mini. I want to schedule Jobs on mini by default, and if it's not enough resources, then proceed to large servers. How can I achieve this goal? Is it possible to set priority for agents(slaves) to specify on which agent I want job to run at first?
The Mesos plugin for Jenkins attempts to build on the most recently built slave (see this method). This means that once it builds on that machine once, as long as that machine still has available spare resources - it'll schedule additional jobs on that machine until it is full. Right now it looks like that isn't optional (I have filed it as a feature request).
Is there any difference between I create two slaves, or one slave with two executors on the same Windows server?
Yes, there is a difference: It's about memory consumption and effort of maintenance/administration.
Starting a slave on a system starts a (main) process. This process costs (private) main memory to run and connects to the master.
Each executor is a sub-process of the main process.
It is therefore apparent that running two executors on one slave costs less memory in total compared to running two slaves (with one executor each), as there would be the memory consumption of the main process twice:
2 * Main Processes + 2 * Executors > 1 * Main Process + 2 * Executors
Moreover, administrating a slave is some more effort than just an executor: Whilst an executor has virtually nothing to worry, there are numerous things to configure for a slave. Additionally, the capabilities of the two slaves are anyhow the same (they are running on the same OS as you said), so there is little value-add to also assign it different labels.
In short, if there are no other boundary conditions, which make me do it differently, I always would prefer running two executors on one slave, as this is easier to administrate and some memory is saved.
A slave is a "machine". An executor is an "OS Process" in the slave.
So ideally we always add executors - they do the work and can run in parallel, and the simple theoretic answer to your question is "2 executors on one slave"
In practice we need to add slaves in several use cases:
We need more resources (more cpu, more memory, more "machines")
We need a different setting (Different OSes, Different hardware)
We have global resources that would create a conflict for executors on same machine (shared browser for a UI testing process)
Make the decision based on your use case.
One benefit which immediately comes to my mind for running 1 executor on given node, is to prevent conflicts between processes run at the same time.
On other hand you could prevent job conflicts using existing Jenkins plugins, ie. Heavy Job, Build Blocker.