Currently I have one big job for a big C++ project, which does everything, compiling, running unit tests, coverage, release binaries and creating docs.
As the job takes 40 minutes I would like to split the job in different smaller ones.
I want to use the following approach:
main job every 15 minutes, which checks out the SCM, compiles the Debug configuration and runs basic unit tests
Several jobs for code analysis, coverage, integration tests, compiling Release builds and deployment to our application server running once per night, if the main job and each previous job were successful
I need the SVN revision, the build number and the workspace of the main job in all following jobs.
So far I was unable to achieve this.
The Parameterize Trigger plugin doesn't support triggers only once a day, the Build Trigger plugin doesn't support parameters, the built-in trigger also didn't work.
I understand that pipelines would probably make my approach easier, but e.g. my used CMake plugin won't support pipelines in a while.
Any other ideas or solutions?
You can just configure a job with parameters (https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build) as post build job, for all your downstream jobs and this plugin:
https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin.
As parameter you can pass any var you need like buildNr and workspace.
Or just have a look at Jenkins Pipeline.
Related
We have a project where we have several Jenkins jobs:
One type of jobs that runs delivery (A),
one that does just compilation and unit tests (B)
and
one that runs integration tests, static code analysis et cetera (C).
We run on four Jenkins nodes (master + three slaves), and our jobs are a mix of declarative pipeline jobs and manually clicked in Jenkins-jobs.
We only want to run one integration test build per node at a time. However, we want to run as many deliveries (A) and code quality (B) builds as there are executors.
Until now, the Throttle concurrent builds (https://github.com/jenkinsci/throttle-concurrent-builds-plugin) plugin has satisfied our needs. However, this plugin does not support declarative pipeline builds, nor does it at all seem updated.
The Lockable resources plugin (https://github.com/jenkinsci/lockable-resources-plugin) seems promising, but we haven't found any way to lock the entire build with a resource-name dynamically set. That is, when we start a C build, we want it to lock "resource_{name of server}".
This plugin allows setting an entire-build-lock in the options directive,
× ut we haven't figured out how to evaluate an environment variable there.
Any suggestions would be highly appreciated!
So the workaround on our side was to rewrite the pipeline script from declarative to scripted syntax. Then, the throttle concurrent builds-plugin works as a charm.
When the bug https://issues.jenkins-ci.org/browse/JENKINS-49173 is fixed, the plugin will work with declarative pipelines as well.
We have set up a Jenkins instance as a remote testing resource for our developers. Every time a tag is created matching our refspec a job is triggered and the results emailed to the developer.
A job is defined as follows:
1 phase consisting of three jobs (frontend tests, integration tests,
unit tests)
All subjobs are executed, irrespective of success
Email the developer the test results
This setup mostly works except for two issues:
I cannot get the job to run in parallel. The subjobs run in
parallel, but only one instance of the job runs at a time. Is this
something I can configure differently somewhere, or is this inherent
in the way the plugin works?
The main job checks out and occupies one of our build servers for
the duration of the job. Is there a way to do git polling and then
just grab the hashref and release the build server on which the
polling was done before continuing building the subjobs?
In the multi job plugin, everything runs in parallel that is listed in the same "Phase", however the multijob itself needs somewhere to run. If you have a build followed by a test phase, you can add a "Build Phase" prior to the test phase, and only that phase will require a "build server".
There is an option called "Execute concurrent builds if necessary" that will allow multiple jobs of the same name to run simultaneously. This option must be set for the parent job and the subjobs as the default behavior of Jenkins is to only allow one build of a Project (job) to run at a time. Beware: Read the comments as this may have unintended side effects.
Not clear what you mean about polling however if using git, you may want to use webhooks so that pushes to the git repository directly invoke Jenkins. No need to poll.
Use case:
CI server polls some VSC repository and runs test suite for each revision. And if two or more revisions were commited, even in a relatively small time interval, I want the CI server to put each of them in queue, run tests for each, store the results, and never run tests again for those commits. And I don't want the CI server to launch jobs in parallel, to avoid performance issues and crashes in case of many simultaneous jobs.
Which CI server is able to handle this?
My additional, less important requirement is that I use Python and it is desirable to use software written in Python, so I looked at the Buildbot project, and I especially want to see reviews for this tool in the matter of is it usable in general and is it capable of replacing most popular solutions like Travis or Jenkins.
I have used jenkins to do this. (with subversion mainly, c/c++ build and also bash/python scripted jobs)
The easiest and default handling of VCS/SCM changes in jenkins is to poll for changes on a set time. A build is triggered if there is any change. More than one commit may be included in build (e.g. if 2 commits are done close together) when using this method. Jenkins shows links back to scm and scm update done as well as showing build logs and you can easily configure build outputs and test result presentation.
https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-Buildsbysourcechanges
What VCS/SCM are you using? Jenkins interfaces to a good few VCS/SCM:
https://wiki.jenkins-ci.org/display/JENKINS/Plugins#Plugins-Sourcecodemanagement
This question answers how to make Jenkins build on every subversion commit:
Jenkins CI: How to trigger builds on SVN commit
TeamCity is free (up to a number of builds and build agents) and feature-rich. It's very easy to install and configure, although it may take some time to find your way through the wealth of options. It is extremely well documented: http://www.jetbrains.com/teamcity/documentation/
It is written in Java but supports many tools natively and others through command-line execution, so you can build anything with it that you want. (I use it mostly for Ruby.) It understands the output of many testing tools; if you're not using one of them maybe yours can emulate their output. It's quite extensible; it has a REST API and a plugin API.
It can be configured to build on each commit, or to build all of the commits that arrived in a given time period, or to trigger in other ways. Docs here: http://confluence.jetbrains.com/display/TCD8/Configuring+VCS+Triggers
By default it starts a single build agent and runs one build at a time on that build agent. You can run more build agents for speed. If you don't want to run more than one build on a machine, only start one build agent on each machine.
I dont want that CI server would launch jobs in parallel to avoid
performance issues and crashes in cases of many simultanious jobs.
In buildbot you can limit the number of running jobs in a salve with max_build parameter or locks
As for Buildbot and Python, you may coordinate parallel builds by configuration, for example:
Modeling Parallel Processes: Steps
svn up
configure
make
make test
make dist
In addition, you can also try using a Triggerable scheduler for your builder which performs steps U,V,W.
From the docs:
The Triggerable scheduler waits to be triggered by a Trigger step (see
Triggering Schedulers) in another build. That step can optionally wait
for the scheduler's builds to complete. This provides two advantages
over Dependent schedulers.
References:
how to lock steps in buildbot
Coordinating Parallel Builds with
Buildbot
There is a Throttle Concurrent Builds Plugin for Jenkins and Hudson. It allows you to specify the number of concurrent builds per job. This is what it says on the plugin page:
It should be noted that Jenkins, by default, never executes the same Job in parallel, so you do not need to actually throttle anything if you go with the default. However, there is the option Execute concurrent builds if necessary, which allows for running the same Job multiple time in parallel, and of course if you use the categories below, you will also be able to restrict multiple Jobs.)
There is also Gitlab CI, a very nice modern Ruby project that uses runners to distribute builds so you could, I guess, limit the number of runners to 1 to get the effect you are after. It's tightly integrated with Gitlab so I don't know how hard it would be to use it as a standalone service.
www.gitlab.com
www.gitlab.com/gitlab-ci
To only run tests once for every revision you can do something like this:
build
post-build
check if the revision of the build is in /tmp/jenkins-test-run
if the revision is in the file skip tests
if the revision is NOT in the file run tests
if we ran the tests then write the ID in /tmp/jenkins-test-run
I have a Jenkins job which compiles and publishes our Java project to a JBoss server. Obviously, the server takes time to start and deploy the new code. I have a second Jenkins job that runs Selenium tests against the running JBoss instance.
I would like to make the second (Selenium) job be performed automatically as a post-build action from the first job (I have already done this), but I want it to be delayed by, say, 2 minutes. The amount of delay time isn't important, but I can't find anywhere that describes how to delay the start of a post-build job. How would I accomplish this?
In the advanced project options of a project configuration, you can set a "quiet period" that does exactly that. Jenkins will wait the specified amount of time after a build has been triggered before actually starting the build.
Alternatively, you could have the JBoss server trigger the build (e.g. by calling a URL) once it's up and running. The advantage of that is what it would take care of cases where the JBoss server doesn't start for some reason.
You might also want to have a look at the Parameterized Trigger Plugin which allows you to run builds of other projects as build steps. This way you could run the Selenium tests as part of the original job and fail if those tests fail.
I need to build and test on multiple configurations: linux, osx and
solaris. I have slave nodes labeled "linux", "osx" and "solaris". On
each configuration, I want to (a) build (b) run smoke tests
(c) if smoke tests pass, then run full tests, and perhaps more.
I thought that multi-configuration jobs might be the answer, so I setup a
multi-configuration build job and it starts concurrent builds on each
OS. The build job will trigger a downstream smoke-test build, which, in
turn, triggers the full-test job.
I've run into the following issues
If one of the configurations fails, the job as a whole fails, and
Jenkins will not fire any downstream jobs (e.g., if the solaris build
fails, Jenkins will not run smoke tests or full tests for osx and
linux).
The solaris build takes about twice as long as the others (on the
order of an hour), and I'd prefer the linux and osx smoke tests not
wait for the solaris build to finish.
Does that mean I'm left with hand-crafting three pipelines of jobs, and
putting them behind a "start-all" job (i.e., creating and hand-chaining
the following jobs)?
build-linux smoke-test-linux full-test-linux
build-osx smoke-test-osx full-test-osx
build-solaris smoke-test-solaris full-test-solaris
Did I miss something obvious?
As far as I know the answer is to create 3 matrix jobs, one for each system. They then would have 3 subjobs (build, smoke-test, fulltest) with the build-job as a touchstone.
Have you thought about combining the build, smoke-test and full tests into a single multi-configuration job? Other than being a little messy, this should work for you.
To answer your first issue: to trigger a downstream job regardless of result, use trigger parameterized build to run when complete (always trigger) and then check "build w/o parameters"
To answer your second issue: either use an all encompassing multi-configuration (matrix) job or use three separate job streams as you mentioned. UPDATE: you could run 3 sequential matrix jobs for each step (build, smoke-test, full tests) but it would mean that if any of the build steps failed then none of the smoke-tests would be run.