I use java -jar jenkins.war and java -jar slave.jar to run Jenkins master and slave. I want to know how many JVM invoked by Jenkins and I can configure the parameter of them.
In Master:
I think only one JVM (I don't run job in master)
In Slave:
java -jar slave.jar => one JVM
every job have a new JVM, this JVM runs pre-build step, get source code (SVN, GIT, ...), post-build step
every Maven will have its own JVM every Junit will have its own JVM
Another question is, I can set the JVM for slave in management node's advanced section, but who use it's configuration?
Every Maven build does not run in its own JVM. Java is multi-threaded. When you launch a slave you can configure the number of threads it can handle, same goes for the master.
Typically you run builds on slave nodes. On unix systems these can be setup to be automatically run from the master on remote nodes.
Manage Jenkins -> Manage Nodes -> New Node
Under Launch Advanced options you can specify the JVM parameters for the remote JVM running the Jenkins node software.
Each option has help, for example the "# executors" option:
This controls the number of concurrent builds that Jenkins can perform. So the value affects the overall system load Jenkins may incur. A good value to start with would be the number of processors on your system.
Increasing this value beyond that would cause each build to take longer, but it could increase the overall throughput, because it allows CPU to build one project while another build is waiting for I/O.
When using Jenkins in the master/slave mode, setting this value to 0 would prevent the master from doing any building on its own. Slaves may not have zero executors, but may be temporarily disabled using the button on the slave's status page.
Related
I have a Jenkins Server (2.204.1) with Docker plugin (1.1.9) and a docker cloud API.
I work with Jenkins docker agents (slaves)
And i map the docker slave build workspace between the container and the host in order to be able to path
Artifacts to the downstream jobs.
in Jenkins Configuration - Docker Cloud Details - Container settings:
Volumes /var/lib/jenkins:/var/lib/jenkins
This works fine for a single build , The problem starts when i run concurrent builds,
They are all mapped to the same workspace on the Docker host and interfering each other.
What would be the best practice when using docker slaves and mapping workspace as a volume ?
I wouldn't like to use $CustomWorkspace or coping artifacts during the build as this is hard to manage and purge.
I prefer the Jenkins regular slave approach of adding #2 to a second concurrent build but this is not the behavior when running concurrent builds on docker slaves
One remote Jenkins agent has no way of knowing whether a given workspace directory is in use by another agent running on the same machine. This is equally true for docker-based agents that share a common directory via volume mounting. Ideally, all agents working from the same machine would have some way of talking to each other to keep from stepping on each other's toes (e.g. a lockfile in the workspace that gets removed upon job termination), but this is not currently the case.
Solution #1: Unique Build Workspaces
If we are using Jenkins pipelines, we can append a unique subdirectory to the workspace directory on a per-build basis. This solution is clean, simple, and easy to implement.
agent {
node {
customWorkspace "${env.BUILD_NUMBER}"
}
}
Ref: https://www.jenkins.io/doc/book/pipeline/syntax/#agent
Solution #2: Unique Agent Workspaces
If this is not possible or desirable, another potential solution is to change the root working directory of the Jenkins agent itself, which can be done by supplying an additional argument to the agent's startup command:
-workDir FILE : Declares the working directory of the
remoting instance (stores cache and logs by
default)
Source: java -jar agent.jar -help
When spinning up multiple agents dynamically on the same machine, we can set this -workDir value to something with a bit more uniqueness to give each agent its own directory to work out of, effectively mitigating workspace collisions. Something like this should work well:
java -classpath agent.jar hudson.remoting.jnlp.Main -headless \
-workDir /var/lib/jenkins/workspace/$(date +%3N) ...
The magic is in the $(date +%3N), which returns the system clock nanoseconds to three digits of precision. We may want to use more or fewer digits because there's a tradeoff: more precision will result in a higher maximum number of workspace directories but decrease the risk of workspace collisions; less precision will have the opposite effect - fewer directories, increased collision risk.
How this command is configured will vary based on your Jenkins setup. For example, we are using the Docker Swarm plugin (v1.9) on Jenkins 2.249.3. Our agent command is configurable at Manage Jenkins >> Manage Nodes and Clouds >> Configure Clouds >> Docker Swarm Cloud Configuration >> Docker Agent templates >> Command.
Ref: https://man7.org/linux/man-pages/man1/date.1.html
I am trying to understand Jenkins distributed builds. From what I have read, Jenkins master uses its' own JVM, and each agent / slave uses its' own JVM.
My Jenkins master is running on a machine with an Ubuntu 16.04 native OS. Am I correct in assuming that each master and agent / slave will have it's own JVM (as oppose to their own linux virtual machines)?
Am I also correct in assuming that once Jenkins master is installed and running, the Jenkins master will orchestrate spinning up the additional JVMs required for the Jenkins agent / slaves?
Thank you for any help,
No, those agents/slaves should run not on the same system (aka VM) as the master. I guess they could, but there wouldn't be any benefits (which is usually: compute power).
If you want more simultaneous builds, add additional machines that have a JVM installed, where the Jenkins Master will launch its slaves (aka agents).
Let's say you have a gitlab instance and it already uses Jenkins for all its CI builds via the gitlab Jenkins plugin, etc. The Jenkins setup has a modest collection of build slaves providing a variety of platforms, etc. and each slave is set up to run just one job at a time (i.e. a Jenkins job gets exclusive access to the build slave, which is important for reasons I won't go into here).
Now let's say you want to consider using gitlab's own native CI support, moving one or more projects over to gitlab instead of Jenkins. The gitlab CI would need to use the same set of build slaves, but it needs to play nice with Jenkins and the two need to cooperate so that if one runs a job on a particular slave, the other won't submit a job to that same slave until the first finishes. In effect, while Jenkins is running a job on a slave, gitlab should see that slave as unavailable and vice versa.
Anyone have working methods for getting gitlab to tell Jenkins it is using a slave while it runs a CI job on there and vice versa? The method doesn't have to be 100% bullet proof, it would potentially be okay if both gitlab and Jenkins run a job on the same slave at the same time if it is a rare event (i.e. race conditions could potentially be tolerated if the frequency of occurrence is likely to be low).
Additional info:
Build slaves include Linux, Windows and Apple.
Docker is not used and would not be permitted at this time.
We have full admin access to everything, but changing code in gitlab or Jenkins themselves would be rejected. Adding scripts or plugins would be okay.
I have many long running jobs that take almost a day to complete. Splitting is not possible. If the network fails then all progress is lost.
How can a slave survive disconnections?
EDIT 1
I have around 300 slaves running in Windows tied to one single Jenkins instance.
Slaves are connected using the manual method java - jar slave.jar -jlnpUrl <serverUrl> <slaveName>. I cannot run them as a regular Windows service because some tests manipulate GUI elements and require a real interactive session otherwise test fail.
EDIT 2
According to Jenkins Cookbook I should be using Cygwin + OpenSSH approach instead of custom script with JLNP-connector. Could this improve stability?
Jenkins was not originally designed for builds to survive across server or slave restarts. There is a CloudBees Long-Running Build plugin that supports long-running builds but, unfortunately, it is available only for enterprise users and still beta.
I didn't find any free alternative and would suggest you to try to improve your network stability and to split your long running jobs. At least you can divide your tests on logical groups (test suites).
Jenkins now has a workflow plugin. It claims to handle "server" restart and loss-of connectivity with slave.
From the link
A key feature of a workflow execution is that it's suspendable. That
is, while the workflow is running your script, you can shut down
Jenkins or lose a connectivity to a slave. When it comes back, Jenkins
will still remember what it was doing, and your workflow script
resumes execution as if it was never interrupted. A technique known as
the "continuation-passing style" execution plays a key role in
achieving this.
(not tested at all)
Edit: Copied from #Jesse Glick's comments :
Workflow is open source and available for anyone running Jenkins 1.580.1+ or later. CloudBees Jenkins Enterprise does include a checkpoint feature, but this is not necessary simply to have a build survive slave disconnections and Jenkins restarts: that is automatic
I am setting up a CI workflow using Jenkins. I have various code bases that I would like to be able to test on different operating systems from Windows Server 2012 through 2003 and also Red Hat, etc.
I'm wondering if using Jenkins slaves would be an effective solution for this.
Specific questions are things like:
If a master executes a project, where is the project defined vs where does the job execute?
If I want to execute a job that tests a language I don't want to support on the masters operating system (think Ruby on Windows), do I still need to make the master aware of that language in order to define the job, say by installing the relevant plugin?
If I define a slave that's running inside a VM and I stop the VM, when the VM comes back up, am I going to have to run some sort of start up task on the slave, or pre-execute task on the master, to re register the slave before I can start a project running on the slave?
When the slave task completes and the results are reported back, are those results stored on the master such that I can shut down the slave and still have access to previous test run results and trending information?
Thanks in advance for any advice.
If a master executes a project, where is the project defined vs where does the job execute?
The jobs are defined and stored on the Master, they are executed on the Slave machines. You can define which jobs get executed on which slaves by using labels.
If I want to execute a job that tests a language I don't want to support on the masters operating system (think Ruby on Windows), do I still need to make the master aware of that language in order to define the job, say by installing the relevant plugin?
The Master doesn't need to know about the build environment. If you set up the Slave with the proper build environment, that should be fine. The master just delegates the jobs and such.
If I define a slave that's running inside a VM and I stop the VM, when the VM comes back up, am I going to have to run some sort of start up task on the slave, or pre-execute task on the master, to re register the slave before I can start a project running on the slave?
It depends on how you are connecting the Slave to the Master. For example, if you connect a Windows machine with the launch method: "Let Jenkins control this Windows slave as a Windows service". It should reconnect automatically when the Slave is back online. There is some setup involved in getting this to work however.
When the slave task completes and the results are reported back, are those results stored on the master such that I can shut down the slave and still have access to previous test run results and trending information?
Console log are kept on the Master. That's probably what you want.
Hope that helps :)