I am trying to scale Jenkins for a large organization. Is there a way to have multiple Jenkins masters share a slave pool? For example, if I had 200 Jenkins Masters and I want them to share the same set of 50 Linux slaves.
That is, assuming each slave only has 1 executor, if Master A submits a job to the slave pool and it is running on Slave 1, if Master B submits a job to the slave pool, it would try to run on one of the other free slaves, since Slave 1 is already occupied.
I know multiple masters could share a single slave if I configured the slave to have a new workspace and executor for each master. However, I want to be able to set the slave up once, instead of having a slave.jar running on the slave for each master.
Cloudbees Op Center appears to provide this functionality, but looking for a way to do this with the Open Source version. If not, how difficult do you think it would be to extend Jenkins to have this functionality? I have Java development experience and have done a little work with Jenkins plugin development.
Thanks,
As you've noted it's not hard to share slaves between masters, just setup multiple workspaces and each master will install it's own slave jar. The trick is to share resources properly.
One such resource manager is Apache Mesos. A Jenkins Mesos plugin exists enabling the creation of slaves on a managed cluster.
This approach is very new and Ebay have blogged on how they've evolved their Jenkins setup to use Mesos:
Ebay CI solutin part 1
Ebay CI solution part 2
Hope this helps.
There is a Gearman Plugin developed by Open Stack to handle sharing slaves by multiple masters.
If it were me, I'd set up all the masters with a cloud plugin for slaves. For example, you could install the kubernetes plugin or the nomad plugin and connect all masters up to the same kubernetes or nomad cluster. Nomad or Kubernetes would take care of resource management, and the masters would only be submitting jobs to a shared pool of resources. This concept can easily be applied to other cloud providers, like AWS, but IMHO if you just want to set up an on-prem pool of resources for your jenkins masters from scratch, nomad is the easiest option.
Related
My requirement is to trigger a CI & CD on a kubernetes on prem infra, whenever a PR has been raised. Jenkins X is an ideal candidate but unfortunately due to few proxy issues it didnt come to fruitition.
Coming to kubernetes-operator, looking for few clarifications.
I've 4 nodes cluster, with one node being the leader.
Do I've to set up a new instance of Jenkins before hand on my K8s cluster or kubernetes-operator does that for me ?
Looking to access the Jenkins instance under the domain : jenkins.mybusinessunit.myorg.com/jenkins
Do I have to do any addtional configurations to enable master - slave set up.
Does kubernetes-operator provides a feature to support CI/CD model like Jenkins X ?
Thanks in advance.
As per your comments you are actually interessted in more of a cloud-native solution to operating a Jenkins, so here goes.
Since you already have a Kubernetes cluster and would like to use the Jenkins Kubernetes operator, then I would recommend you use the Jenkins Kubernetes Plugin for managing your workloads.
The Jenkins Kubernetes plugin enables you to run each of your pipelines in a separate Pod in your Kubernetes cluster, and once the required Service resources are setup, then the communication between master and slave pods is completely regulated by the plugin. I would recommend that you look into their documentation, which is quite good ( in comparison to other plugins ).
Now since you are also using the Jenkins Kubernetes operator you should know that the plugin is installed as one of the default plugins and is available as soon as your Jenkins instance is spun up. I would recommend you read through the Jenkins Kubernetes operator documentation to get a better grasp of what happens when that is running.
So now I will move onto your questions.
Do I've to set up a new instance of Jenkins before hand on my K8s cluster or kubernetes-operator does that for me ?
If you install the Jenkins Kubernetes operator via Helm chart, then no, you get a Jenkins master instance included. Otherwise if you install the controller into your cluster manually, then you will need to create a Jenkins CRD, which will create a Jenkins instance for you.
Looking to access the Jenkins instance under the domain : jenkins.mybusinessunit.myorg.com/jenkins
Use Ingress + Load Balancer + DNS Service or expose the Pod via NodePort. Note that exposing your master Pod via NodePort may require you to make your Jenkins Master instance publicly available ( and that may not be wise ).
Do I have to do any addtional configurations to enable master - slave set up.
Please refer to the documentation of Jenkins Kubernetes plugin and Jenkins Kubernetes operator. All details are provided there, but configuration is rather minimal.
Does kubernetes-operator provides a feature to support CI/CD model like Jenkins X
No. The Jenkins Kubernetes operator is there only to manage your Jenkins instance and backups in immutable fashion. Jenkins X can be used in combination with Jenkins, but neither replaces the other completely.
I am trying to implement CI/CD pipeline by using Kubernetes and Jenkins. I am planning to use Kubernetes HA Cluster having 3 master and 5 worker machine / node.
Now I am exploring about the implementation tutorials for CI/CD Pipeline. And also exploring about the Jenkins usage with Kubernetes HA Cluster. When I am reading , I felt little bit confusions about Jenkins. That I am adding here.
1. I have total 8 VMs - 3 Master and 5 Worker machines / nodes (Kubernetes cluster). If I installing Jenkins in any one worker machines , then is there any problem while integrating with CI/CD pipeline for deployment ?
2. I am previously readed the following link for understanding the implementations,
https://dzone.com/articles/easily-automate-your-cicd-pipeline-with-jenkins-he
Is this mandatory to use Jenkins master and slave ?. In this tutorial showing that If kubectl,helm and docker is installed then don't need to use Jenkins slave. What is the idea about master and slave here?
3. If I am installing both jenkins master and slave in kubernetes cluster worker machine / node , then Need to install master and slave in separate separate VMs? I have still confusion about where to install Jenkins?
I am just started on CI/CD pipeline - Kubernetes and Jenkins.
Jenkins has two parts. There's the master, which manages all the jobs, and the workers, which perform the jobs.
The Jenkins master supports many kinds of workers (slaves) via plugins - you can have stand alone nodes, Docker based slaves, Kubernetes scheduled Docker slaves, etc.
Where you run the Jenkins master doesn't really matter very much, what is important is how you configure it to run your jobs.
Since you are on Kubernetes, I would suggest checking out the Kubernetes plugin for Jenkins. When you configure the master to use this plugin, it will create a new Kubernetes pod for each job, and this pod will run the Docker based Jenkins slave image. The way this works is that the plugin watches for a job in the job queue, notices there isn't a slave to run it, starts the Jenkins slave docker image, which registers itself with the master, then it does the job, and gets deleted. So you do not need to directly create slave nodes in this setup.
When you are in a Kubernetes cluster in a container based workflow, you don't need to worry about where to run the containers, let Kubernetes figure that out for you. Just use Helm to launch the Jenkins master, then connect to the Jenkins master and configure it to use Kubernetes slaves.
I'm starting to work with Jenkins , I'm a newbie, and I have a question regarding Master-slave configuration. How's usually used? Is it one slave per application (i.e: 3 applications, 3 slaves)?
PS: If you point me to a good Jenkins tutorial, will be appreciate it
First, my recommendation is to disable all the executors on the master and to build only on slaves.
On our Jenkins platform, we are using 7 CentOS slaves (VM) for the Maven builds (one executor per node).
These slaves have the same Jenkins label ("build linux" for example).
All the Maven projects are restricted to this "build linux" label:
https://serverfault.com/questions/359793/tell-jenkins-to-run-a-specific-project-on-a-particular-slave-node
We also have dedicated slaves per platform (Linux 32bits, Linux 64bits, AIX 6, AIX 7, Windows 2008 R2, Windows 2012).
If you need to deploy your applications on dedicated Application Servers (Tomcat, Weblogic, ...), you can create one slave per platform.
Using Jenkins label is really useful to distribute your builds on the relevant slaves.
First recommendation is to use at least one slave and not use the master for building.
Read https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds (first answer when googling "jenkins master slave tutorial").
Then, add as much slaves as you want to distribute the load. The load does not directly depends on the number of "applications" but on the number of simultaneous builds and available slaves to perform those builds (you may have multiple jobs for a unique application). Typically, you will add slaves if your builds are "pending" for too much time: see https://wiki.jenkins-ci.org/display/JENKINS/Executor+Starvation
Tie the jobs to labels rather than directly to the slaves.
You will obviously have different slaves per build environment.
If your build is isolated enough and it is not multi-threaded, then it is also recommended to set about as much executors as core CPU. That is to make use of the slave resources; another solution is to use virtualization, containers (VM, docker...) as slaves with one executor per slave.
I'm new to Jenkins, and I like to know if it is possible to have one Jenkins server to deploy / update code on multiple web servers.
Currently, I have two web servers, which are using python Fabric for deployment.
Any good tutorials, will be greatly welcomed.
One solution could be to declare your web servers as slave nodes.
First thing, give jenkins credentials to your servers (login/password or ssh login+private key or certificate. This can be configured in the "Manage credentials" menu
Then configure the slave nodes. Read the doc
Then, create a multi-configuration job. First you have to install the matrix-project plugin. This will allow you to send the same deployment intructions to both your servers at once
Since you are already using Fabic for deployment, I would suggest installing Fabric on the Jenkins master and have Jenkins kick off the Fabric commands to deploy to the remote servers. You could set up the hostnames or IPs of the remote servers as parameters to the build and just have shell commands that iterate over them and run the Fabric commands. You can take this a step further and have the same job deploy to dev/test/prod just by using a different set of hosts.
I would not make the webservers slave nodes. Reserve slave nodes for build jobs. For example, if you need to build a windows application, you will need a windows Jenkins slave. IF you have a problem with installing Fabric on your Jenkins master, you could create a slave node that is responsible for running Fabric deploys and force anything that runs a fabric command to use that slave. I feel like this is overly complex but if you have a ton of builds on your master, you might want to go this route.
I've been reading about Jenkins master/slave configurations but I still have some questions:
Is it so that the slave Jenkins is not actually installed and started up the way master Jenkins is? I assumed I would install one master Jenkins and another slave Jenkins in the same way, and then master Jenkins would control the slave e.g. through SSH? So I cannot view the slave Jenkins through a GUI?
The reason why I have thought about adding a slave Jenkins on another VM is because the VM contains our application servers (many test environments). Deploying and starting/stopping application servers from master Jenkins is a pain because master Jenkins and application servers are on different machines. Therefore, if I would add a slave Jenkins to the machine where our application servers are, these would actually be deployed and started/stopped locally (by slave Jenkins). I wonder if I have missed something, of if my presumptions are still valid.
In a standard Jenkins master/slave setup, Jenkins is only installed on the master. That is where you see the user interface and start/configure build jobs.
The slaves execute the jobs. There is no Jenkins installation here other than a small Java app to have Jenkins communicate to/from the slave. Jenkins talks to these slaves through the slave.jar app over e.g. SSH via the SSH Slaves Plugin and can monitor if the slave is running, etc.
So in your case, you can start jobs from the master that will execute on the application servers.
The master/slave setup also allows you to host all whole bunch of different slaves, with different OSes, different hardware, etc. You can communicate job results (artifacts) from one slave to another via the Copy Artifacts Plugin.
There are also ways to duplicate the actual Jenkins master with load balancing in a heavy use scenario. That is not what you seem to be looking for.