I am creating a list of Jenkins jobs for sanity test of our Jenkins build environment. I want to create layers of jobs. The first layer of jobs will check the environment, e.g. if all slaves are up, the 2nd layer then can check the integration to other tools such as GitHub, TFS, SonarQube, then the 3rd layer can run some typical build projects. This sanity test can also be used to verify the environment after any major changes to the Jenkins servers.
We have about 10 slaves created on two servers, one Windows and one Linux. I know I can create a job to run on a specific slave, therefore test if the slave is online, but this way I need to create 10 jobs just to test all slaves. Is there a best approach to check if all slaves are online?
One option is to use Jenkins Groovy scripting for a task like this. The Groovy plugin provides the Jenkins Script Console (a useful way to experiment) and the ability to run groovy scripts as build steps. If you're going to use the Script Console for periodic maintenance, you'll also want the Scriptler plugin which allows you to manage the scripts that you run.
From Manage Jenkins -> Script Console, you can write a groovy script that iterates through the slaves and checks whether they are online:
for (node in Jenkins.instance.nodes) {
println "${node.name}, ${node.numExecutors}"
def computer = node.computer
println "Online: ${computer.online}, ${computer.connectTime} (${computer.offlineCauseReason})"
}
Once you have the basic checks worked out, you can create either a standalone script in Scriptler, or a special build to run these checks periodically.
It often takes some iteration to figure out the right set of properties to examine. As I describe in another answer, you can write functions to introspect the objects available to scripting. And so with some trial and error, you can develop a script performs the checks you want to run.
Related
What I'd like to do:
Before/after any build allocates a slave: Reset slave to a pristine state (eg restore/delete files, kill rogue processes, etc. In other words, run some script). I know how to do this in one job, I don't know how to have Jenkins trigger it for every job of every type without modifying them all.
Unsatisfactory Ways I can think of accomplishing this:
Enforce a pre/post step in every AbstractProject (old style jobs). This is only scalable if using the Job DSL Plugin to generate all jobs.
Enforce a pre/post step in every PipelineJob. This is only scalable if using a Jenkins shared library (eg: myBuild() that encapsulates these steps).
Make use of the Global Post Script plugin and run a groovy script after every build. This works, but the script runs after the slave has been released, so another build may have grabbed it already (it's too late). Figuring out what nodes were allocated during the build is also fairly complicated
Switch all slaves to one-shot type (ie Docker) that require no clean-up. This doesn't work for my use cases, but may work for someone else.
Periodically run a job that uses a System Groovy Script to edit all other jobs to add a "clean-up" "pre" build step if not present. (Will not work for Pipeline jobs)
(I currently do #1 and #2)
Ideal Solution
Theoretical options:
A plugin to hook into some event (ie WorkspaceListener.beforeUse() ) and execute something then (unfortunately, WorkspaceListener does not apply to Pipeline Jobs). This should trigger right before a slave is used. It gets a little complicated when a slave has multiple executors (mine don't).
A plugin to enforce execution of some steps, similar to "Execute builders from job X", but in every job. (also doesn't work for Pipeline jobs)
Assuming that I'm using Swarm for all slaves, modify the swarm client to handle this logic (perform a task when slave becomes idle). A poor man's way would be to make the swarm client run in "one-shot" mode, in a bash loop.
Question:
What am I overlooking? Is there a better way?
I want to have one Jenkins job control the build number of another job but without the inconvenience of reloading the entire project configuration from disk. I have seen that it's easily possible to directly update the nextBuildNumber file of the target job (I can do this as a build step of Job A) but this does not take effect immediately. Restarting Jenkins or even reloading the Jenkins configs from disk takes way too long and can only be done when there are no builds in progress.
I have tried the groovy script mentioned in the below post by running it from the Manage Jenkins > Script Console. The same post also suggests the script can be saved as a file and run from the CLI. Can it be run from a build step?
I want Job A to determine Job B's next build number and set it so that Job B can run (later in the same day) with the desired build number.
https://stackoverflow.com/a/20077362/4306857
Perhaps I should clarify. I'm not familiar with Groovy so I'm looking at the various build step options like "Execute Windows batch command" which I have a lot of experience with. I can see an "Invoke Gradle script" option so I was wondering if there may be a plugin that can run Groovy scripts perhaps?
The reason this requirement has arisen is that we are compiling a product for two different platforms. We want to compile the codebase almost simultaneously for both platforms with two jobs (A & B) which will both update the JIRA cases included in the builds. We feel it will be less confusing to have both these jobs running with the same build number so that when we talk about a particular issue being addressed in build #75, say, we won't have to qualify that by stating the full job name. If JOB-A build #75 and JOB-B build #75 are both compiled on the same day from the same codebase we can test and compare the results of both builds with far less confusion than if the build numbers go out of sync.
Obviously, in the short term we will use the Set Next Build Number plugin to manually keep the build numbers in step but we want to automate this if possible.
Depends on whether or not you are using Version Number plugin:
[ X ] Create a formatted version number
Build Display Name [ X ] Use the formatted version number for build display name.
Assuming you are NOT, this groovy script will do:
def NextNumber=42
job=Jenkins.instance.getItemByFullName('path/to/jobName')
job.nextBuildNumber = NextNumber
job.save();
You will need groovy plugin for that. Place that in an "Execute system Groovy script" step. Make sure to choose system groovy. That will execute on the master, where the job config and metadata is stored so you have access to the Jenkins internals and data.
I'd suggest you should really be using the above options rather than relying on "keeping both jobs in sync" via a script or manually. You can then pass the label to be used from the first job as a parameter to the second job. That would also require Parameterized Trigger as well as Version Number plugins.
You can even use ${BUILD_DATE_FORMATTED} or ${BUILD_TIMESTAMP}, etc.
Postdate: thinking about the problemspace from a different perspective, that of running 2+ builds on different platforms (simultaneously), there's a plugin for that: Matrix project. You can run it as a freeatyle job on multiple nodes or is excellently described as Matrix building in scripted pipeline. Not sure how that would tie in to JIRA.
I'm searching for a way to execute automatically a global configured script BEFORE a Jenkins job will be started.
My use case is, all Jenkins jobs are only allowed to start if a specific environment variable is set.
If a variable is not set, the build should be aborted.
I found the Global Post Plugin https://wiki.jenkins.io/display/JENKINS/Global+Post+Script+Plugin, i only need the oposite what this Plugin does.
Maybe there's another solution?
I needed to chmod my /data/jenkins/.npm and /data/jenkins/.sbt directories before running all my builds.
I could either add a prebuild step to every job (redundant and messy) or I could go under Manage Jenkins -> Configure System.
We have a Cloud -> Amazon EC2 configuration section with "Init script" - you can add what you want to run there on slave startup.
However, if you really want something to run something for every job (not enough to run on jenkins slave startup) then you probably don't want to manually configure it for each job.
I suggest you look into Jenkins DSL as you can define preBuildSteps section on any/all job(s) which can then reference a common snippet (eg. a shell script to run).
Partial Solution:
Take a look at the Global Pre Script plugin. This plugin is less feature-rich than the Global Post Script plugin, but it should do at least a part of what you want. It notably lacks the option to abort the build, but it is able to manipulate parameters or other preconditions that your jobs rely on. You may also be able to submit a PR to add some means of preventing the build from executing.
Some options:
Modify Global Pre Script to be able to cleanly abort the build from groovy.
Change your existing jobs to check for a precondition (manually or via script). This not the most scalable option.
Replace your existing jobs with Pipeline jobs and use Shared Libraries to bottleneck the logic. (This is what I do).
Generate your jobs using the Job DSL Plugin and enforce a pre build step in every generated job. (This is what I also do)
Limitations:
Something to keep in mind for both global plugins: neither plugin provides a proper build step. The groovy code executes on the master.
One use case that neither plugin will handle is a between-job slave cleanup/sanity check.
I would like to clean up the list of JDKs managed by our Jenkins CI server.
For instance there are several versions of JDK 8 whereas I'd like to only have one of them (the latest). So I am looking for a simple way for identifying all those Jenkins jobs explicitly using one of the other JDK installations so I can change them to use the one remaining JDK install.
Is there a better way to do so apart from manually clicking through all jobs and examine which JDK they use?
I often use the Jenkins Script Console for tasks like this. The Groovy plugin provides the Script Console, but if you're going to use the Script Console for periodic maintenance, you'll also want the Scriptler plugin which allows you to manage the scripts that you run.
From Manage Jenkins -> Script Console, you can write a groovy script that iterates through the jobs:
for (job in Jenkins.instance.items) {
println job.name
...examine more details of job...
}
It often takes some iteration to figure out the right set of job properties to examine. It's not clear to me how your jobs are configured, so you'll have to figure out what property describes the JDK used by the job. The Change JVM Options script might provide some hints.
As I describe in another answer, you can write functions to introspect the objects available to scripting. And so with some trial and error, develop a script that walks the list of jobs and examines the properties that you're interested in.
Use case:
CI server polls some VSC repository and runs test suite for each revision. And if two or more revisions were commited, even in a relatively small time interval, I want the CI server to put each of them in queue, run tests for each, store the results, and never run tests again for those commits. And I don't want the CI server to launch jobs in parallel, to avoid performance issues and crashes in case of many simultaneous jobs.
Which CI server is able to handle this?
My additional, less important requirement is that I use Python and it is desirable to use software written in Python, so I looked at the Buildbot project, and I especially want to see reviews for this tool in the matter of is it usable in general and is it capable of replacing most popular solutions like Travis or Jenkins.
I have used jenkins to do this. (with subversion mainly, c/c++ build and also bash/python scripted jobs)
The easiest and default handling of VCS/SCM changes in jenkins is to poll for changes on a set time. A build is triggered if there is any change. More than one commit may be included in build (e.g. if 2 commits are done close together) when using this method. Jenkins shows links back to scm and scm update done as well as showing build logs and you can easily configure build outputs and test result presentation.
https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-Buildsbysourcechanges
What VCS/SCM are you using? Jenkins interfaces to a good few VCS/SCM:
https://wiki.jenkins-ci.org/display/JENKINS/Plugins#Plugins-Sourcecodemanagement
This question answers how to make Jenkins build on every subversion commit:
Jenkins CI: How to trigger builds on SVN commit
TeamCity is free (up to a number of builds and build agents) and feature-rich. It's very easy to install and configure, although it may take some time to find your way through the wealth of options. It is extremely well documented: http://www.jetbrains.com/teamcity/documentation/
It is written in Java but supports many tools natively and others through command-line execution, so you can build anything with it that you want. (I use it mostly for Ruby.) It understands the output of many testing tools; if you're not using one of them maybe yours can emulate their output. It's quite extensible; it has a REST API and a plugin API.
It can be configured to build on each commit, or to build all of the commits that arrived in a given time period, or to trigger in other ways. Docs here: http://confluence.jetbrains.com/display/TCD8/Configuring+VCS+Triggers
By default it starts a single build agent and runs one build at a time on that build agent. You can run more build agents for speed. If you don't want to run more than one build on a machine, only start one build agent on each machine.
I dont want that CI server would launch jobs in parallel to avoid
performance issues and crashes in cases of many simultanious jobs.
In buildbot you can limit the number of running jobs in a salve with max_build parameter or locks
As for Buildbot and Python, you may coordinate parallel builds by configuration, for example:
Modeling Parallel Processes: Steps
svn up
configure
make
make test
make dist
In addition, you can also try using a Triggerable scheduler for your builder which performs steps U,V,W.
From the docs:
The Triggerable scheduler waits to be triggered by a Trigger step (see
Triggering Schedulers) in another build. That step can optionally wait
for the scheduler's builds to complete. This provides two advantages
over Dependent schedulers.
References:
how to lock steps in buildbot
Coordinating Parallel Builds with
Buildbot
There is a Throttle Concurrent Builds Plugin for Jenkins and Hudson. It allows you to specify the number of concurrent builds per job. This is what it says on the plugin page:
It should be noted that Jenkins, by default, never executes the same Job in parallel, so you do not need to actually throttle anything if you go with the default. However, there is the option Execute concurrent builds if necessary, which allows for running the same Job multiple time in parallel, and of course if you use the categories below, you will also be able to restrict multiple Jobs.)
There is also Gitlab CI, a very nice modern Ruby project that uses runners to distribute builds so you could, I guess, limit the number of runners to 1 to get the effect you are after. It's tightly integrated with Gitlab so I don't know how hard it would be to use it as a standalone service.
www.gitlab.com
www.gitlab.com/gitlab-ci
To only run tests once for every revision you can do something like this:
build
post-build
check if the revision of the build is in /tmp/jenkins-test-run
if the revision is in the file skip tests
if the revision is NOT in the file run tests
if we ran the tests then write the ID in /tmp/jenkins-test-run