I'm new to Jenkins and Continous Integration, and I noticed that it supports master / slave nodes. I really don't know what that means.
Can someone please tell in what scenario do I need slave agent?
Here is a scenario:
Our main Jenkins master is running on Windows machine (yes I know... I know...). We are doing iOS mobile development. There are some things that can only be done using Xcode (which only runs on Mac OS). I have a Jenkins Slave running on that Mac, that takes care of executing those tasks that can only run on a Mac.
Why not just setup a new instance on that Mac? Cause that job is tied together with other jobs (on Master) in dependencies and the flow. Even promotions on those Xcode tasks are run on Master.
Jenkins' Master / Slave architecture is used to manage distributed builds.
There are many different scenarios you might want to use a distributed build system. It is all based on your projects load and dependencies.
Pretty much, the Master is what you're probably currently using, and is responsible for scheduling builds, dispatching jobs to slaves, and monitoring the results, but can also execute jobs itself. A slave is a java executable that sits on a remote server waiting for instructions from the master (to execute build).
To use this functionality in Jenkins, go to "Manage Jenkins" screen, and click on "Manage Nodes"
https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds
For a more in depth summary of distributed builds with Jenkins, as well as some scenarios where this system would be useful, and how to implement it, please read chapter 11: Distributed Builds of Jenkins: The Definitive Guide by John Ferguson Smart
http://wakaleo.com/books/jenkins-the-definitive-guide/download-jtdg-pdf
Related
I have a 4-5 java projects build procedure to be configured as CI using Jenkins. Whether it will be time saving to build some/all projects on different machines(connected as Jenkins slaves)?
Are there some any other benefits of Jenkins Master-Slave configuration?
Offloading work to build agents is a very good idea as it keeps load away from the master. This allows you to build more projects in parallel (esp. with dynamic agents launched in some cloud environment).
Further, it makes the systems easier to maintain, as e.g. the Java version/setup of your build agents required to build and test your application does not interfere with the Jenkins master machine.
I have many long running jobs that take almost a day to complete. Splitting is not possible. If the network fails then all progress is lost.
How can a slave survive disconnections?
EDIT 1
I have around 300 slaves running in Windows tied to one single Jenkins instance.
Slaves are connected using the manual method java - jar slave.jar -jlnpUrl <serverUrl> <slaveName>. I cannot run them as a regular Windows service because some tests manipulate GUI elements and require a real interactive session otherwise test fail.
EDIT 2
According to Jenkins Cookbook I should be using Cygwin + OpenSSH approach instead of custom script with JLNP-connector. Could this improve stability?
Jenkins was not originally designed for builds to survive across server or slave restarts. There is a CloudBees Long-Running Build plugin that supports long-running builds but, unfortunately, it is available only for enterprise users and still beta.
I didn't find any free alternative and would suggest you to try to improve your network stability and to split your long running jobs. At least you can divide your tests on logical groups (test suites).
Jenkins now has a workflow plugin. It claims to handle "server" restart and loss-of connectivity with slave.
From the link
A key feature of a workflow execution is that it's suspendable. That
is, while the workflow is running your script, you can shut down
Jenkins or lose a connectivity to a slave. When it comes back, Jenkins
will still remember what it was doing, and your workflow script
resumes execution as if it was never interrupted. A technique known as
the "continuation-passing style" execution plays a key role in
achieving this.
(not tested at all)
Edit: Copied from #Jesse Glick's comments :
Workflow is open source and available for anyone running Jenkins 1.580.1+ or later. CloudBees Jenkins Enterprise does include a checkpoint feature, but this is not necessary simply to have a build survive slave disconnections and Jenkins restarts: that is automatic
I am setting up a CI workflow using Jenkins. I have various code bases that I would like to be able to test on different operating systems from Windows Server 2012 through 2003 and also Red Hat, etc.
I'm wondering if using Jenkins slaves would be an effective solution for this.
Specific questions are things like:
If a master executes a project, where is the project defined vs where does the job execute?
If I want to execute a job that tests a language I don't want to support on the masters operating system (think Ruby on Windows), do I still need to make the master aware of that language in order to define the job, say by installing the relevant plugin?
If I define a slave that's running inside a VM and I stop the VM, when the VM comes back up, am I going to have to run some sort of start up task on the slave, or pre-execute task on the master, to re register the slave before I can start a project running on the slave?
When the slave task completes and the results are reported back, are those results stored on the master such that I can shut down the slave and still have access to previous test run results and trending information?
Thanks in advance for any advice.
If a master executes a project, where is the project defined vs where does the job execute?
The jobs are defined and stored on the Master, they are executed on the Slave machines. You can define which jobs get executed on which slaves by using labels.
If I want to execute a job that tests a language I don't want to support on the masters operating system (think Ruby on Windows), do I still need to make the master aware of that language in order to define the job, say by installing the relevant plugin?
The Master doesn't need to know about the build environment. If you set up the Slave with the proper build environment, that should be fine. The master just delegates the jobs and such.
If I define a slave that's running inside a VM and I stop the VM, when the VM comes back up, am I going to have to run some sort of start up task on the slave, or pre-execute task on the master, to re register the slave before I can start a project running on the slave?
It depends on how you are connecting the Slave to the Master. For example, if you connect a Windows machine with the launch method: "Let Jenkins control this Windows slave as a Windows service". It should reconnect automatically when the Slave is back online. There is some setup involved in getting this to work however.
When the slave task completes and the results are reported back, are those results stored on the master such that I can shut down the slave and still have access to previous test run results and trending information?
Console log are kept on the Master. That's probably what you want.
Hope that helps :)
I would like to raise from scratch a build server for .NET applications using Jenkins, please note that i'm new to Jenkins CI.
Several Questions:
1) How should I decide on the build server specs? except for the OS which would be windows server 2012, how should I decide on the RAM and the CPU and HD space?
2) Should the Jenkins sit in the build machine or not, what is the recommended approach? I understood that the build server should be isolated from the Jenkins master
3) How do I decide on the Master/Slave approach, when should I use only Master and when should I use master and slave or slaves?
4) How would you recommend me to run the build and deployment tasks in the Jenkins CI, using NAnt/Python or any other scripting language ?
10x, and sorry for the igonrance :)
Responding to each in turn:
You can run Jenkins as a windows service (instructions here) and the machine can be a VM, so it doesn't have to be huge.
a) RAM and CPU: I'll put these together and will depend on how many jobs you plan to have running at the same time. The default number of build executors is 3 but can be increased as a global config change.
b) HDD: This depends on how many jobs you plan to have. Jenkins will checkout the source code (as well as the compiled output) to its home directory on a per job basis. This can get big. I would also recommend using the ThinBackup plugin to backup the Jenkins configuration.
Jenkins is the build machine. A vanilla installation of Jenkins is the master. In my experience you will not need a separate slave machine unless you're needing to do native builds on other platforms or have LOTS and LOTS of jobs. I've seen single masters running happily with hundreds of jobs.
Further to 2. above, suggest you start with a master and set up a slave later if you really need one.
As you have stated you are building .NET applications, you can simply install the MSBuild plugin which should serve you well. Builds for .NET applications in Jenkins are Freestyle builds so you will be using Windows Batch build steps often as well. This also is a great blog on Jenkins in a .NET environment.
I am using Jenkins for CI,
I've heard that I should have a dedicated server and slave for running Jenkins and building tasks, respectively -
is this true?
I can understand this as the server may not be powerful enough to handle the server itself and running build tasks,
but is there any defined technical reason for this?
Best practice is to have a separate machine for Jenkins-Server,
and not to use it for builds at all.
This has nothing to do with CPU-power or memory-resources -
A build-machine should have a predefined configuration,
and Jenkins should not be part of it.
(Jenkins requirements may even conflict with those of the build-machine)
You should be able to boot / clone / upgrade / restore / trash the build-machine
without any impact on Jenkins.
Of course you can settle for a single machine, if your resources are limited,
but if you are serious about build-automation - Jenkins should have its own server.
You probably don't need dedicated hardware/VM to run a Jenkins server because the actual Jenkins process (no builds running) uses very little resources. But it all depends on what you want to accomplish with your Jenkins setup.
Do you want to run continuous builds across multiple platforms for multiple projects? Then using a master with slaves is the only way to go. If, on the other hand, you're running fairly simple builds for just a few projects, then you only need one machine to run the builds and the Jenkins process.
You can configure Jenkins to have multiple builds running concurrently so if you have a quad-core machine, you can safely run 2 builds and possibly a third once you analyze resource usage.
At my last gig, I used a quad-core machine with 8GB RAM to run:
Jenkins running Selenium builds
VirtualBox VM with Windows XP
Two instances of Tomcat each with two applications deployed.
And the machine still had more to spare.