Use case:
CI server polls some VSC repository and runs test suite for each revision. And if two or more revisions were commited, even in a relatively small time interval, I want the CI server to put each of them in queue, run tests for each, store the results, and never run tests again for those commits. And I don't want the CI server to launch jobs in parallel, to avoid performance issues and crashes in case of many simultaneous jobs.
Which CI server is able to handle this?
My additional, less important requirement is that I use Python and it is desirable to use software written in Python, so I looked at the Buildbot project, and I especially want to see reviews for this tool in the matter of is it usable in general and is it capable of replacing most popular solutions like Travis or Jenkins.
I have used jenkins to do this. (with subversion mainly, c/c++ build and also bash/python scripted jobs)
The easiest and default handling of VCS/SCM changes in jenkins is to poll for changes on a set time. A build is triggered if there is any change. More than one commit may be included in build (e.g. if 2 commits are done close together) when using this method. Jenkins shows links back to scm and scm update done as well as showing build logs and you can easily configure build outputs and test result presentation.
https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-Buildsbysourcechanges
What VCS/SCM are you using? Jenkins interfaces to a good few VCS/SCM:
https://wiki.jenkins-ci.org/display/JENKINS/Plugins#Plugins-Sourcecodemanagement
This question answers how to make Jenkins build on every subversion commit:
Jenkins CI: How to trigger builds on SVN commit
TeamCity is free (up to a number of builds and build agents) and feature-rich. It's very easy to install and configure, although it may take some time to find your way through the wealth of options. It is extremely well documented: http://www.jetbrains.com/teamcity/documentation/
It is written in Java but supports many tools natively and others through command-line execution, so you can build anything with it that you want. (I use it mostly for Ruby.) It understands the output of many testing tools; if you're not using one of them maybe yours can emulate their output. It's quite extensible; it has a REST API and a plugin API.
It can be configured to build on each commit, or to build all of the commits that arrived in a given time period, or to trigger in other ways. Docs here: http://confluence.jetbrains.com/display/TCD8/Configuring+VCS+Triggers
By default it starts a single build agent and runs one build at a time on that build agent. You can run more build agents for speed. If you don't want to run more than one build on a machine, only start one build agent on each machine.
I dont want that CI server would launch jobs in parallel to avoid
performance issues and crashes in cases of many simultanious jobs.
In buildbot you can limit the number of running jobs in a salve with max_build parameter or locks
As for Buildbot and Python, you may coordinate parallel builds by configuration, for example:
Modeling Parallel Processes: Steps
svn up
configure
make
make test
make dist
In addition, you can also try using a Triggerable scheduler for your builder which performs steps U,V,W.
From the docs:
The Triggerable scheduler waits to be triggered by a Trigger step (see
Triggering Schedulers) in another build. That step can optionally wait
for the scheduler's builds to complete. This provides two advantages
over Dependent schedulers.
References:
how to lock steps in buildbot
Coordinating Parallel Builds with
Buildbot
There is a Throttle Concurrent Builds Plugin for Jenkins and Hudson. It allows you to specify the number of concurrent builds per job. This is what it says on the plugin page:
It should be noted that Jenkins, by default, never executes the same Job in parallel, so you do not need to actually throttle anything if you go with the default. However, there is the option Execute concurrent builds if necessary, which allows for running the same Job multiple time in parallel, and of course if you use the categories below, you will also be able to restrict multiple Jobs.)
There is also Gitlab CI, a very nice modern Ruby project that uses runners to distribute builds so you could, I guess, limit the number of runners to 1 to get the effect you are after. It's tightly integrated with Gitlab so I don't know how hard it would be to use it as a standalone service.
www.gitlab.com
www.gitlab.com/gitlab-ci
To only run tests once for every revision you can do something like this:
build
post-build
check if the revision of the build is in /tmp/jenkins-test-run
if the revision is in the file skip tests
if the revision is NOT in the file run tests
if we ran the tests then write the ID in /tmp/jenkins-test-run
Related
What I'd like to do:
Before/after any build allocates a slave: Reset slave to a pristine state (eg restore/delete files, kill rogue processes, etc. In other words, run some script). I know how to do this in one job, I don't know how to have Jenkins trigger it for every job of every type without modifying them all.
Unsatisfactory Ways I can think of accomplishing this:
Enforce a pre/post step in every AbstractProject (old style jobs). This is only scalable if using the Job DSL Plugin to generate all jobs.
Enforce a pre/post step in every PipelineJob. This is only scalable if using a Jenkins shared library (eg: myBuild() that encapsulates these steps).
Make use of the Global Post Script plugin and run a groovy script after every build. This works, but the script runs after the slave has been released, so another build may have grabbed it already (it's too late). Figuring out what nodes were allocated during the build is also fairly complicated
Switch all slaves to one-shot type (ie Docker) that require no clean-up. This doesn't work for my use cases, but may work for someone else.
Periodically run a job that uses a System Groovy Script to edit all other jobs to add a "clean-up" "pre" build step if not present. (Will not work for Pipeline jobs)
(I currently do #1 and #2)
Ideal Solution
Theoretical options:
A plugin to hook into some event (ie WorkspaceListener.beforeUse() ) and execute something then (unfortunately, WorkspaceListener does not apply to Pipeline Jobs). This should trigger right before a slave is used. It gets a little complicated when a slave has multiple executors (mine don't).
A plugin to enforce execution of some steps, similar to "Execute builders from job X", but in every job. (also doesn't work for Pipeline jobs)
Assuming that I'm using Swarm for all slaves, modify the swarm client to handle this logic (perform a task when slave becomes idle). A poor man's way would be to make the swarm client run in "one-shot" mode, in a bash loop.
Question:
What am I overlooking? Is there a better way?
Following the Jenkins Best Practices, I want to avoid that Build Jobs/Pipelines could be executed into my Jenkins Master.
To do so, I've installed the Job Restrictions Plugin, using it to configure the Master to run only some Maintenance Pipelines.
The problem is that now Build Pipelines that are configured to run on specific Agents, are not executed anymore. I see that the Build Queue continuously grows, and the Pipelines are not runned. I think that this behaviour could be related to Flyweight Executors of the Master.
So, the question is the following: How can I execute on Master just a little subset of Maintenance Pipelines and, in the mean time, execute Build Pipelines only on specific Agent?
You can configure the master node to only be used when explicitly named. Just click the master node > go to configure and change Use this node as much as possible to Only build jobs with label expressions matching this node
I found the solution that perfectly fits with my needs, here.
To quickly sum up the solution, I was to able to exclude all the user Builds from Master and run on it only the Jobs/Pipelines of a specific Jenkins folder (IuA in my case), configuring the Job Restrictions Plugin in the following way:
In order to better understand the logic behind this solution, I recommend you to give a look at link that I posted above.
I want to have one Jenkins job control the build number of another job but without the inconvenience of reloading the entire project configuration from disk. I have seen that it's easily possible to directly update the nextBuildNumber file of the target job (I can do this as a build step of Job A) but this does not take effect immediately. Restarting Jenkins or even reloading the Jenkins configs from disk takes way too long and can only be done when there are no builds in progress.
I have tried the groovy script mentioned in the below post by running it from the Manage Jenkins > Script Console. The same post also suggests the script can be saved as a file and run from the CLI. Can it be run from a build step?
I want Job A to determine Job B's next build number and set it so that Job B can run (later in the same day) with the desired build number.
https://stackoverflow.com/a/20077362/4306857
Perhaps I should clarify. I'm not familiar with Groovy so I'm looking at the various build step options like "Execute Windows batch command" which I have a lot of experience with. I can see an "Invoke Gradle script" option so I was wondering if there may be a plugin that can run Groovy scripts perhaps?
The reason this requirement has arisen is that we are compiling a product for two different platforms. We want to compile the codebase almost simultaneously for both platforms with two jobs (A & B) which will both update the JIRA cases included in the builds. We feel it will be less confusing to have both these jobs running with the same build number so that when we talk about a particular issue being addressed in build #75, say, we won't have to qualify that by stating the full job name. If JOB-A build #75 and JOB-B build #75 are both compiled on the same day from the same codebase we can test and compare the results of both builds with far less confusion than if the build numbers go out of sync.
Obviously, in the short term we will use the Set Next Build Number plugin to manually keep the build numbers in step but we want to automate this if possible.
Depends on whether or not you are using Version Number plugin:
[ X ] Create a formatted version number
Build Display Name [ X ] Use the formatted version number for build display name.
Assuming you are NOT, this groovy script will do:
def NextNumber=42
job=Jenkins.instance.getItemByFullName('path/to/jobName')
job.nextBuildNumber = NextNumber
job.save();
You will need groovy plugin for that. Place that in an "Execute system Groovy script" step. Make sure to choose system groovy. That will execute on the master, where the job config and metadata is stored so you have access to the Jenkins internals and data.
I'd suggest you should really be using the above options rather than relying on "keeping both jobs in sync" via a script or manually. You can then pass the label to be used from the first job as a parameter to the second job. That would also require Parameterized Trigger as well as Version Number plugins.
You can even use ${BUILD_DATE_FORMATTED} or ${BUILD_TIMESTAMP}, etc.
Postdate: thinking about the problemspace from a different perspective, that of running 2+ builds on different platforms (simultaneously), there's a plugin for that: Matrix project. You can run it as a freeatyle job on multiple nodes or is excellently described as Matrix building in scripted pipeline. Not sure how that would tie in to JIRA.
Currently I have one big job for a big C++ project, which does everything, compiling, running unit tests, coverage, release binaries and creating docs.
As the job takes 40 minutes I would like to split the job in different smaller ones.
I want to use the following approach:
main job every 15 minutes, which checks out the SCM, compiles the Debug configuration and runs basic unit tests
Several jobs for code analysis, coverage, integration tests, compiling Release builds and deployment to our application server running once per night, if the main job and each previous job were successful
I need the SVN revision, the build number and the workspace of the main job in all following jobs.
So far I was unable to achieve this.
The Parameterize Trigger plugin doesn't support triggers only once a day, the Build Trigger plugin doesn't support parameters, the built-in trigger also didn't work.
I understand that pipelines would probably make my approach easier, but e.g. my used CMake plugin won't support pipelines in a while.
Any other ideas or solutions?
You can just configure a job with parameters (https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build) as post build job, for all your downstream jobs and this plugin:
https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin.
As parameter you can pass any var you need like buildNr and workspace.
Or just have a look at Jenkins Pipeline.
I have many long running jobs that take almost a day to complete. Splitting is not possible. If the network fails then all progress is lost.
How can a slave survive disconnections?
EDIT 1
I have around 300 slaves running in Windows tied to one single Jenkins instance.
Slaves are connected using the manual method java - jar slave.jar -jlnpUrl <serverUrl> <slaveName>. I cannot run them as a regular Windows service because some tests manipulate GUI elements and require a real interactive session otherwise test fail.
EDIT 2
According to Jenkins Cookbook I should be using Cygwin + OpenSSH approach instead of custom script with JLNP-connector. Could this improve stability?
Jenkins was not originally designed for builds to survive across server or slave restarts. There is a CloudBees Long-Running Build plugin that supports long-running builds but, unfortunately, it is available only for enterprise users and still beta.
I didn't find any free alternative and would suggest you to try to improve your network stability and to split your long running jobs. At least you can divide your tests on logical groups (test suites).
Jenkins now has a workflow plugin. It claims to handle "server" restart and loss-of connectivity with slave.
From the link
A key feature of a workflow execution is that it's suspendable. That
is, while the workflow is running your script, you can shut down
Jenkins or lose a connectivity to a slave. When it comes back, Jenkins
will still remember what it was doing, and your workflow script
resumes execution as if it was never interrupted. A technique known as
the "continuation-passing style" execution plays a key role in
achieving this.
(not tested at all)
Edit: Copied from #Jesse Glick's comments :
Workflow is open source and available for anyone running Jenkins 1.580.1+ or later. CloudBees Jenkins Enterprise does include a checkpoint feature, but this is not necessary simply to have a build survive slave disconnections and Jenkins restarts: that is automatic