How do you maintain the Jenkins job configuration in SCM along side the source code?
As source code evolves, so does the job configuration. It would be ideal to be able to keep the job configuration in SCM, for the following benefits:
easy to see who a history of the changes, including the author and the description
able to rebuild old branch/tag by checking out the revision and build just work
not having to scroll through the UI to find the appropriate section and make change
I see there is a Jenkins Job Builder plugin. I prefer a solution along the lines of Travis CI, where the job configuration is maintained in a YAML file (.travis.yml). Any good suggestions?
Note: Most of our projects are using Java & Maven.
Update 2016: Jenkins now provides a Jenkinsfile which provides exactly this. This is supported by the core Jenkins developers and actively developed.
Benefits:
Creating a Jenkinsfile, which is checked into source control, provides a number of immediate benefits:
Code review/iteration on the Pipeline
Audit trail for the Pipeline
Single source of truth for the Pipeline, which can be viewed and edited by multiple members of the project.
I've written a plugin that does this!
Other than my plugin, you have some (limited) options with existing Jenkins plugins:
Use a single test script
If you configure your Jenkins to simply run:
$ bash run_tests.sh
You can then check in a run_tests.sh file into your SCM repo and you're now tracking changes for how you run tests. However, this won't track configuration of any plugins.
Similarly, if you're using Maven, the Maven Project Plugin simply runs a specified goal for your repo.
The Literate Plugin does allow Jenkins to run the commands in your README.md, but it hasn't yet been released.
Track changes to Jenkins configuration
You can use the SCM Sync configuration plugin to write configuration changes to SCM, so you at least have a persistent record. This is global, across all projects on your Jenkins instance.
There's also the job config history plugin, which stores config history on the filesystem.
Write Jenkins configuration from SCM
The Jenkins job builder project you mentioned lets you check config changes into SCM and have them applied to your Jenkins instance. Again, this is across all projects on your Jenkins instance.
Write Jenkins configuration from another job
You can use the Job DSL Plugin with a repo of groovy scripts. Jenkins then polls that repo, executes the groovy scripts, which create job configurations.
Discussions
Issue 996 (now closed) discusses this, and it has also been discussed on the mailing list: 'Keeping track of Hudson's configuration changes', and 'save hudson config in svn'.
you can do this all with the workflow plugin and a lot more. Workflow is one of the most advanced technics to use jenkins and it has a very strong support.
It is based on a groovy DSL and allows you to keep the whole configuration in the SCM of your choise (e.g. GIT, SVN...).
Related
I'm in the process of migrating git-based projects to use a shared Pipeline definition from a governance tier built with Jenkins Templating Engine.
In the process of testing I cloned the project and pushed it to a new repository in Bitbucket where it was recognized by Jenkins and the template was used immediately based on the definitions in pipeline_config.groovy. However, this is not a sane migration path for existing projects. How do I get Jenkins to start using the template on branches without Jenkinsfile and the Jenkinsfile on branches with a Jenkinsfile.
The result of of "Scan Multibranch Pipeline Now" according to the logs is ‘Jenkinsfile’ not found. Skipping. I assume that a new project regonizer is provided by the Jenkins Templating Plugin.
I assume that every project with Git Flow has to perform this migration, so I'm confused there's no documentation.
I'm using Jenkins 2.306 and JTE plugin 2.3.
It seems that newer versions of JTE or Jenkins allow a mixture of branches with JTE and without.
In case you have to deal with versions that don't do the following:
Once I removed Jenkinsfile from every branch and put a pipeline_config.groovy on every branch the Multibranch Plugin started recognizing the project first as removed and at the next scan as present with all branches which were all using the Jenkins Template Engine.
Not the best migration imo, but a great opportunity to cleanup old branches. Since my project was using Git Flow I needed to make a technical hotfix release to also migrate away from Jenkinsfile on master.
I am currently using the Jenkins supermarket cookbook to deploy an instance of Jenkins running as a service on my Chef managed node. So far I have modified my _master_war_ recipe file to install Jenkins and start Jenkins as a service using runit, install all plugin versions I need as well as enable matrix based security and create a base administrator account. It should also be noted we are using BitBucket for source control.
I also want to add my jobs as part of the automated Jenkins deployment. From what I understand, the way to do this is to copy the job configuration files from a directory on my chef workstation to a directory (Chef's cache path), on the managed node.
Per the Jenkins public supermarket cookbook readme:
The :create action requires a Jenkins job config.xml. This config file
must exist on the target node and contain a valid Jenkins job
configuration file. Because the Jenkins CLI actually reads and
generates its own copy of this file, do NOT write this configuration
inside of the Jenkins job. We recommend putting them in Chef's file
cache path.
As these job configurations do change periodically, I'm wondering what is the best way to maintain the most recent copy of the job configuration file(s) on my chef workstation for deployment to my managed node?
Am I understanding the cookbook documentation correctly in that we will need a local copy of the job configuration file (on the Chef workstation), that is then copied to Chef's cache file path on the managed node?
Thanks in advance for any help anyone is able to provide.
Personally, I count setting up Jenkins jobs a lot more into the domain of Jenkins instead of Chef. The Jenkins community has developed several "jobs as code" approaches, the most popular ones being the Job DSL and the Jenkins Pipelines, with the latter one being the probably better starting point.
What remains up to the Chef cookbook is to define the seed job, either e.g. for the "Bitbucket Organisation Folder" plugin (and the one job that points to your organisation at BitBucket) or a so-called "Seed job" for the Job DSL.
Regarding automated setup of pipelines, I recommend a look at Torben Knerr's examples. This uses
Pipeline (as defined in Jenkinsfiles) for the actual build jobs
a Job DSL seed job to set up the pipeline job(s)
Regarding the actual implementation in Chef, you can see an example in a cookbook of mine. The template resource copies a file from the cookbook (in the templates/ subdirectory) into some temporary path, from where the jenkins_job resource picks it up (on the Jenkins server).
So I'm not sure, if you got it right regarding:
will need a local copy of the job configuration file (on the Chef workstation)
So you just need it once on your workstation to add it to the cookbook, yes.
I am trying to understand how to best deploy an instance of Jenkins, complete with plugins, users and jobs using Chef. I am currently using the Chef Jenkins Supermarket cookbook.
I am attempting to achieve automated deployment of our Pipelines as part of the project. From what I have gathered, the best way to go about this is to have Chef configure a seed job in Jenkins initial setup and configuration.
The seed job should specify, among other things, the git repository from which to find and use a Jenkinsfile for a given job. I've found this resource by Daniel Spilker to be helpful in explaining seed jobs.
So the seed Jenkins job would be run, which would then generate the Jenkins job we have just scripted with it (in this case the seed job would be to pull the Jenkinsfile from source control and configure a new Jenkins job (our pipeline), with the details of the Jenkinsfile).
Am I understanding this correctly as the proper way to not only automate Jenkins job configuration, but also as the proper way to always have an up to date job configuration for any given job in the event the job configuration were to change?
If we used a seed job to setup our pipeline, what are some possible solutions to having the initial seed job run automatically once Jenkins is fully configured by Chef?
As for job configuration changes that may occur over time, would we need to setup the seed job to poll source control for any changes in the Jenkinsfile periodically in the event the Jenkinsfile has been modified? (It may be helpful to note that we are currently using BitBucket for source control).
Just getting started with pipeline as code. Thanks to everybody in advance for their patience and guidance.
I've mentioned this a bit in your other questions, but the least painful approach is to treat Jenkins as a database, not a web service. Have Chef do the basic install, but then configure the initial bits by hand. For DR, rely on your backups rather than Chef.
I have 15 Jenkins jobs configured in order to implement a specific flow. I am improving and editing these jobs as time goes by.
Is there a way to save all these jobs and its configurations to a repository or at least in the form of export jobs, save it and import when needed?
There are two plugins that will help you save Jenkins jobs, "SCM Sync configuration" and "JobConfigHistory" defined at the wiki.jenkins-ci.org website.
SCM Sync Configuration Plugin (which keeps the config in a SCM repository)
or
Job Config History Plugin (Saves copies of all job and system configurations)
The Job DSL Plugin allows to define the jobs in a DSL and store the DSL scripts in a SCM repo. The DSL increases the readability of the config files in contrast to the XML format.
For an intro, see the slides and video from the Configuration as Code: The Job DSL Plugin talk at the Jenkins User Conference 2015 in London.
You can move/copy jobs to another destination simply by copying directory with job (default path for jobs directories is /var/lib/jenkins/jobs).
You can get more info here - https://wiki.jenkins-ci.org/display/JENKINS/Administering+Jenkins
The Workflow feature may let you write your entire process as one (Groovy) script, which you can then maintain in your version control system alongside other sources.
We have a long life cycle for our software releases. That we need a continuous integration server also for our branches.
The problem is that we have the sources of the software in a repository but not the jobs of the Jenkins server. One problem is that the jobs for a branch use other repository URLs. But also many other settings of the jobs can change over the years.
Ideally the configuration of the CI server would be also branched.
Is it possible to start any CI server with a configuration in a repository?
Is it possible to use placeholders in the repository URL that will be replaced with the branch of the CI configuration?
See the SCM Sync configuration plugin:
SCM Sync Configuration Jenkins/Hudson plugin is aimed at 2 main features:
Keep sync'ed your config.xml (and other ressources) jenkins/hudson files with a SCM repository
Track changes (and author) made on every file with commit messages