I am trying to understand Jenkins distributed builds. From what I have read, Jenkins master uses its' own JVM, and each agent / slave uses its' own JVM.
My Jenkins master is running on a machine with an Ubuntu 16.04 native OS. Am I correct in assuming that each master and agent / slave will have it's own JVM (as oppose to their own linux virtual machines)?
Am I also correct in assuming that once Jenkins master is installed and running, the Jenkins master will orchestrate spinning up the additional JVMs required for the Jenkins agent / slaves?
Thank you for any help,
No, those agents/slaves should run not on the same system (aka VM) as the master. I guess they could, but there wouldn't be any benefits (which is usually: compute power).
If you want more simultaneous builds, add additional machines that have a JVM installed, where the Jenkins Master will launch its slaves (aka agents).
Related
I'm starting to work with Jenkins , I'm a newbie, and I have a question regarding Master-slave configuration. How's usually used? Is it one slave per application (i.e: 3 applications, 3 slaves)?
PS: If you point me to a good Jenkins tutorial, will be appreciate it
First, my recommendation is to disable all the executors on the master and to build only on slaves.
On our Jenkins platform, we are using 7 CentOS slaves (VM) for the Maven builds (one executor per node).
These slaves have the same Jenkins label ("build linux" for example).
All the Maven projects are restricted to this "build linux" label:
https://serverfault.com/questions/359793/tell-jenkins-to-run-a-specific-project-on-a-particular-slave-node
We also have dedicated slaves per platform (Linux 32bits, Linux 64bits, AIX 6, AIX 7, Windows 2008 R2, Windows 2012).
If you need to deploy your applications on dedicated Application Servers (Tomcat, Weblogic, ...), you can create one slave per platform.
Using Jenkins label is really useful to distribute your builds on the relevant slaves.
First recommendation is to use at least one slave and not use the master for building.
Read https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds (first answer when googling "jenkins master slave tutorial").
Then, add as much slaves as you want to distribute the load. The load does not directly depends on the number of "applications" but on the number of simultaneous builds and available slaves to perform those builds (you may have multiple jobs for a unique application). Typically, you will add slaves if your builds are "pending" for too much time: see https://wiki.jenkins-ci.org/display/JENKINS/Executor+Starvation
Tie the jobs to labels rather than directly to the slaves.
You will obviously have different slaves per build environment.
If your build is isolated enough and it is not multi-threaded, then it is also recommended to set about as much executors as core CPU. That is to make use of the slave resources; another solution is to use virtualization, containers (VM, docker...) as slaves with one executor per slave.
I've been reading about Jenkins master/slave configurations but I still have some questions:
Is it so that the slave Jenkins is not actually installed and started up the way master Jenkins is? I assumed I would install one master Jenkins and another slave Jenkins in the same way, and then master Jenkins would control the slave e.g. through SSH? So I cannot view the slave Jenkins through a GUI?
The reason why I have thought about adding a slave Jenkins on another VM is because the VM contains our application servers (many test environments). Deploying and starting/stopping application servers from master Jenkins is a pain because master Jenkins and application servers are on different machines. Therefore, if I would add a slave Jenkins to the machine where our application servers are, these would actually be deployed and started/stopped locally (by slave Jenkins). I wonder if I have missed something, of if my presumptions are still valid.
In a standard Jenkins master/slave setup, Jenkins is only installed on the master. That is where you see the user interface and start/configure build jobs.
The slaves execute the jobs. There is no Jenkins installation here other than a small Java app to have Jenkins communicate to/from the slave. Jenkins talks to these slaves through the slave.jar app over e.g. SSH via the SSH Slaves Plugin and can monitor if the slave is running, etc.
So in your case, you can start jobs from the master that will execute on the application servers.
The master/slave setup also allows you to host all whole bunch of different slaves, with different OSes, different hardware, etc. You can communicate job results (artifacts) from one slave to another via the Copy Artifacts Plugin.
There are also ways to duplicate the actual Jenkins master with load balancing in a heavy use scenario. That is not what you seem to be looking for.
Travis CI has a really nice feature, builds are run within VirtualBox VMs. Each time a build is started, the box is refreshed from a snapshot and the code copied on to it. Any problems with the build cannot affect the host, and you can use any OS to run your builds on.
This would be really good, for example, compiling and testing code on a guest OS that matches your production env. Also, you can keep your host free of any installation dependencies you might need (e.g. a database server) and run ITs without worrying about things like port conflicts.
Does such a thing exist for Jenkins?
Check out the Vagrant Plugin https://wiki.jenkins-ci.org/display/JENKINS/Vagrant-plugin
This plugin allows booting of Vagrant virtual machines, provisioning them and also executing scripts inside of them
You can run Jenkins in a Master Slave Setup. Your Master instance manages all the jobs but lets all the slaves do the actual work. These Slaves can be VMs or physical machines. Go To Manage Jenkins -> Manage Nodes -> New Node to add Nodes to your Jenkins Setup.
There is the vSphere Cloud Plugin and the Scripted Cloud Plugin that can be used for this purpose.
I have setup lots of Windows Masters with Windows Slaves.
I have instructions for setting up a Linux Master.
However, what I would really like to do is to setup a Linux (ubuntu) Slave on a Windows master.
The issue is that my Windows master doesn't have a SSH option for slaves, so none of the Linux Slave instructions seem to apply.
Any ideas if this is possible and if so, how to go about it?
Frank
There shouldn't be any major differences between running a master on different operating systems.
I thought the SSH Slaves plugin came installed by default, but if not, you can try installing it from the plugin centre.
I can only build my system on a FreeBSD 5 machine.
I am looking and introducing gerrit and Jenkins into my team.
I have setup the server running gerrit and Jenkins both on a Debian machine for now.
What is the standard way of dealing with Jenkins building on a remote server?
The standard way would be to install a Jenkins Slave on the remote server. See https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds
You then setup the job that must be built on a FreeBSD 5 machine, to only only run on that slave/node. This is can be achieved within the job configuration by specifying the FreeBSD machine by name.
If you don't want other Jenkin jobs to run on that FreeBSD machine, you can configure the slave to only run jobs that are tied to it. See step 4 in the JENKINS Step by step guide to setup master and slave machines for the config page that you need to do this.