I have
a usual jenkins master jettysburg:8888/jenkins,
and a failover jenkins master jettyperry:8888/jenkins.
I am planning to use mercurial to keep the jobs directory of the two masters in sync.
Usually, most of the time, new jobs and builds are defined and executed at jettysburg:8888. Then I would need to sync jettyperry:8888 with whatever took place in jettysburg:8888, and I plan to perform the push once a day.
Which means, after a failover to jettyperry:8888, the push would be performed the converse direction.
I have been using mercurial for my non-programming/coding files (like word, excel and text files) for that purpose, as a means to perform incremental and redundant backup of my "mission critical" files, as well as versioning them.
I am also hoping to depend on mercurial to backout from changes made to jenkins jobs.
Is using mercurial to sync two Jenkins masters a good idea? Is there a better way to keep two Jenkins servers in sync? In this case, I am syncing only the jobs tree.
As long as you don't need shared job state between the servers (they run in their own little universes) and you keep the same plugin modules and libraries on both Jenkins servers, using some form version control to keep the actual job definitions is fine.
My office does this with git. We have a development and a production set of jenkins servers. We maintain a base linux image with jenkins install with all necesarry modules and locally install libraries (like nodejs and such). We then spin up an instance of the image and pull down the jobs.
The one thing that can be a challenge is keeping in sync things like credentials and Jenkins config settings--you might need to keep them as part of the base image.
If you need the job queues to persist and be shared (like a master-master setup) you can look at this plugin, which allows multiple jenkins masters to share the same job queue: https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
There are two approaches to this:
Instead of approaching a solution with hot back-up why not consider clustered master so you get an active-active solution. You may want to look at https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin. This helps you cluster a bunch of masters so one going down should not be an issue.
Consider running Jenkins on containers, and externalize jenkins projects directory as external volumes in an NFS so you can bring up another container when one goes down - keeping both containers running will be a challenge with concurrent writes (if there are any).
Hope this helps.
Related
I'm aware I'm lacking basic Jenkins concepts but with my current knowledge it's hard for to research successfully - maybe you can give me some hints I can use to re-word my question if needed.
Currently I'm facing a situation in which in a setup with several build nodes the Jenkins master machine is running out of disk space because Jenkins clones git repositories on both, the master and build nodes (and the master only has limited space). This question explains why.
Note: the master node itself does not build anything - it just clones the repo to a local workspace folder (I guess it just needs the Jenkinsfiles).
Going through the job configurations and googling this issue I find options regarding shallow and sparse clones or cleaning up the workspace before or after the build using the Cleanup Plugin. But those settings and plugins only care about the checkout done with checkout(SCM) on the build nodes, not the master.
But in case I want to leave the situation as is on the build nodes but keep the workspace folders on the Jenkins master machine slim, how do I approach this? What do I have to search for?
And as a side question - isn't it possible to have something like "git exports"? I.e. having the .git folders removed after checking out the commit I need?
In case it depends on the kind of job I use, I'm using scripted pipeline jobs.
I've got a similar setup: A master node, multiple build nodes.
Simply, I set the number of executors=0 on the master node (from Manage Jenkins -> Manage Nodes), so every job will land on build nodes.
The only repo cloned on the master is the shared library.
Running Jenkins builds in the master node is discouraged for two main reasons:
First of all, the usability of the Jenkins platform might be affected by many ongoing builds, for example showing delays on certain operations.
It is a well-known security problem, as pointed out by the documentation:
Any builds running on the built-in node have the same level of access to the controller file system as the Jenkins process.
It is therefore highly advisable to not run any builds on the built-in node, instead using agents (statically configured or provided by clouds) to run builds.
Always in that wiki page you can find details on this security problems, like what an attacker can do and an alternative that lets you use the master node to build, but patching some of the listed security problems. The solution is based on a plugin called Job Restrictions Plugin.
By the way, the most popular decision is to let slave nodes do the build:
To prevent builds from running on the built-in node directly, navigate to Manage Jenkins » Manage Nodes and Clouds. Select master in the list, then select Configure in the menu. Set the number of executors to 0 and save. Make sure to also set up clouds or build agents to run builds on, otherwise builds won’t be able to start.
If you really have strong reasons to build on the master node, you can always apply a different git clone strategy based on the value of the env.NODE_NAME environment variable. It is set to master if the pipeline job is run on the master node, otherwise it is filled with the node name (of course). Nonetheless, I have never seen anyone customizing the git clone command based on the node used, so... Don't do it 😉
About the sparse checkout and the sparse/shallow clone:
The former creates an incomplete working directory, avoiding to map all the trees and blobs present in the current commit, but only those you specify. Do you save that much space? Or better, is your project tree that heavy that you would need to do something like this? The sparse-checkout is generally used when you want a clean working tree, without unnecessary files.
sparse/shallow clone can be useful sometimes to reduce the download time, especially when you have a huge history. The most common option is --depth=1 that instructs git to retrieve only the most recent commit. As far as I know, Jenkins already applies some optimizations to speed the clone process but it generally keeps the entire history. Again, I am not sure you would gain a lot more space.
A valid (at least for me) alternative to space-optimizations on git files, is to build on Docker containers. Jenkins has reached a good level of integration with Docker and there are a lot of advantages using it, among which the disposal of the workspace after the job finished.
I didn't use the pipeline feature myself so far -- but conceptually it is clear that the master requires initial access to the Jenkinsfile. It will therefore be difficult to avoid this step entirely.
If Jenkins itself does not provide an option to fine-tune the clone/checkout behavior on the master side, then I'd see these options:
Create a custom version of Jenkins (or of the corresponding plugin) which hard-codes the behavior that you need (like, shallow/sparse clone). Modifying and building both Jenkins and its plugins is surprisingly simple; often, the most difficult part is to locate the code that you need to touch.
Tune the master's clone in-place. Shallowness and sparse-checkout properties can be set for existing clones. If you set these properties after the initial clone (possibly in the Jenkinsfile itself or in a post-build step), then Jenkins may possibly maintain those properties.
Constantly re-cloning and deleting the repo on master side increases the load both on the Jenkins master and on your Git server, so better be careful with that (especially since your repository has a size where disk space matters already). If you really want to go that way, you could try to force-remove the clone on the master in a post-build step -- this should be relatively easy to implement. You need to check that this hack will not interfere with Jenkins' access to the Jenkinsfile.
Why is Jenkins changing node to do another build when it doesn't need to?
We have a Jenkins setup with 3 Mac and 3 Windows nodes, building free style projects. We do not use the master for builds. The projects are set to run on any of the 3 nodes suitable for their platform. We are using labels for this purpose.
Some of the time, when we do a build it will do the build on the same node as last time.
But sometimes, without any obvious pattern, it will change to a different node, even though the previously used node is available and not busy. This wastes a lot of time potentially as incremental builds on the same node are much faster than pulling and building everything from scratch.
Jenkins claims it should allocate jobs to the same node if possible, when multiple possibilities exist.
As a result, from the user’s point of view, it looks as if Jenkins tries to always use the same node for the same job, unless it’s not available, in which case it’ll build elsewhere. But as soon as the preferred node is available, the build comes back to it. Reference
Jenkins version is 2.289.2 presently.
The jobs are freestyle builds with shell script/command prompt steps.
Repositories are Git and Mercurial.
I think if you read the Restrict where this project can be run that might provide you answer to your question.
As its definition tells By default, builds of this project may be executed on any agents that are available and configured to accept new builds.
So if you don't specify the agent and restrict the build to be run a particular node, it will randomly choose.
This wastes a lot of time potentially as incremental builds on the
same node are much faster than pulling and building everything from
scratch.
To solve the above problem only, we have the Restrict where this project can be run feature in Jenkins. Please look to the below screenshot for detailed help that we can get in Jenkins .
I have recently started to mess about with Jenkins and am unsure how to deploy my web app to a basic server. I've gotten into the Pipeline (https://jenkins.io/doc/book/pipeline/) and it seems like a fantastic way to work.
Where I'm a bit stuck is in two spots:
Once my repo is in my workspace within Jenkins, how do I prep it so I am only deploying the files necessary for the application? For example, I don't need my src/ directory or my Vagrantfile when I'm deploying things.
How do I deploy my app to the server? I see examples all over the place, but I am getting a bit lost since there seems to be so many ways to do this. I'm assuming scp or something like that...?
To build off of #2, is there a way to deploy web apps as transactions (in one shot) rather than file-by-file?
Please let me know if I can provide any information for potential answers!
I can't speak to your specific use case but a common way to do this is the build-and-deploy model, where you will have 2 Jenkins jobs. The "build" job will check out from source, run build commands such as maven or make, and lastly will "archive" the build artifacts. The latter is an option under the 'post-build actions' tab at the bottom.
In the "deploy" job, you will grab the artifacts of your choice. You can fetch a single file, all of them, and everything in between. This requires use of the 'Copy Artifact' plug-in and it allows you to copy files generated by other jobs. Now you can run your usual deploy script in the 'Execute Command' box. Most command line paradigms are supported out of the box such as setting environment variables.
The instructions above assume that you want to run your application off of a host that you've provisioned as a Jenkins slave.
Use artifacts as mentioned by Paul Back, or a 3rd party artifactory server as in video
This is always tricky and error-prone. Why not spin up a fresh server with new release (humanly verified once)
Jenkins & Ansible is the answer here. This is how I deploy to production, since I am in no need to use anything like Docker (too many issues with particular app) so have to run the app natively. Quick example would be
You monitor a specific branch in gitlab / github or whatever else and then call a webhook on push / merge etc on that branch, at this point you deal with anything you need to do by running a playbook on the jenkins job that monitors that branch (jenkins).
in my case jenkins and ansible run on the same server. Jenkins runs the ansible playbook that does whatever I need to do.
for example with ansible, I copy certain files that need to be there, run configs / change filenames etc. setup nginx, run composer,
you get the point.
I am trying to wrap my head around this. Most CI/CD examples/projects have a single master that is always released, and have some variant of, e.g. git-flow, to have a develop branch. Once tagged, it goes to master.
Either way, master is always released to production.
But in the real world as I see it, there are human gates for release to production and other environments. What mechanism do you use to manage the deployment of different versions?
For example:
v1.5 is the current production release
v1.6 has passed all tests, artifacts are ready, it is tagged as valid, but business decides to deploy it only to staging, awaiting an opportune moment to deploy
v1.5 is deployed to a demo environment
v2.0 has also passed all tests, but is in UAT, subject to the customer being happy, as it is a major release
There could be many more such environments - production, staging, UAT, demo, demo2, etc.
What mechanism do you use to handle the tagging of a particular version for a particular environment, and the actual deployment thereof?
Although there a probably a few ways to do it, I use the build pipeline plugin https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin Along with the copy artifacts plugin https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin
With these, you can create individual jobs for each piece of your environment, and link them altogether.
So as in your example, the pipeline would look like:
Build -> Test and Deploy to UAT (2.0) -> deploy to staging(1.6) -> demo(1.5) -> prod (1.5)
Each piece represents a different build in jenkins. The idea behind continuous integration is you create the binaries once, and you carry it down the pipeline, only changing configuration pieces along the way. In a build job, the artifacts are created and then archived. In any jobs after, the artifact is picked up from the upstream job, some stuff is done, and then it get's re-archived for the next downstream job. So the deploy to staging would go to the Test and Deploy to Uat job to get its binary. The entire concept of Continuous Delivery boils down to the the build pipeline. http://en.wikipedia.org/wiki/Continuous_delivery (and yes I did just cite wikipedia).
As for tagging individual binaries for specific environments, that is by definition, not continuous integration. A binary is suppose to be created in a way that it can easily be propagated from one environment to the next. So unfortunately, individual builds for specific environments can never be continuous delivery. You can use jenkins as a CI server all you want, but if your process does not match, you will never achieve true continuous integration.
Braching, merging and checkins always seems to be a touchy subject when it comes to Continuous Integration, so I won't go into it much. But a lot of people share the idea that : "If different members of the team are working on separate branches, then by definition, they not participating in continuous integration process." http://eugenedvorkin.com/continuous-integration-strategies-for-branching-and-merging/
EDIT
For Flagging specific builds, it sounds like your looking to take use of this feature : https://wiki.jenkins-ci.org/display/JENKINS/Fingerprint ... Which gets the job done effectively, giving you the entire life of any individual artifact. A bit more complex solution would be artifactory, which is essentially artifact source control.
I explained the concept of the deployment process above, and without information on your specific environment it is hard to go much further. But for me, for java applications deployed to tomcat containers, the deploy plugin works great https://wiki.jenkins-ci.org/display/JENKINS/Deploy+Plugin
You shouldn't have to worry about selection of which artifact to deploy. The pipeline should be setup to always deploy the latest artifact that was archived in its corresponding upstream job.
Maybe Docker can help you out with this issue. It is able to deploy images of projects to a specific environment. If that environment has a docker client or a docker deamon you are able to request specific information about that environment and the project (to be) deployed on it.
Jenkins can still play a huge part in your pipeline for the integration part and you could let docker do the delivery part.
Docker: https://www.docker.com
Docker plugin for jenkins: https://wiki.jenkins-ci.org/display/JENKINS/Docker+build+step+plugin
Docker also has support for windows machines and .NET.
I want to deploy a private cloud test infrastructure using OpenStack and Jenkins for multiple projects. I thought of creating a template for OpenStack with one Jenkins installation using as master. For the projects I thought of separating them into nodes, i.e. each project would get one node. Is this a sensible structure? Or should I install one Jenkins installtion per project+vm?
1) How would you organize a private multi-project test cloud infrastructure?
2) Jenkins saves configuration and job information to /var/lib/jenkins by default, how do I manage the object storage for each project?
When you say node, I'm assuming you mean a machine running nova-compute and hosting VM instances. If this is the case, then I honestly wouldn't worry about trying to bind a project to a specific node - treat the entire openstack pool of resources you have as a global cluster, assign in projects, and let them spin up and tear down as they need.
You will likely find it beneficial to have an image with jenkins pre-installed as a publically available image, assuming you want a master jenkins per project in your cloud. If you're running jenkins as a stand-alone item per project, using a m1.medium may be sufficient, but you might find you want to use m1.large. It all depends on what you have your jenkins instance doing in each project.
If you want the jenkins data to persist across destroying and recreating the jenkins master instance, then you could use a volume and specifically mount /var/lib/jenkins into it - but you will need to manage the coordination of jenkins startup and having the volume attached appropriately. You may find it easier to give the jenkins instance a larger base disk and just back up and restore the data per project if you need to destroy and recreate the jenkins instance.
Another way to do this would be to share a master jenkins external to your openstack cloud and use the jclouds jenkins plugin to spin up jenkins instances and slaves as you need for projects. This isn't providing any segregation between projects in jenkins, which may not be to your liking based on the question above.