What ideally I need to do is run a Jenkins pipeline that will run immediately after I start the execution , checking out code, running a build but just running the deployment in a particular date/time. So Would it be possible to schedule a particular step in jenkins pipeline? This step would be executed at date/time provided by user input. If that’s not possible is there any alternative way of doing it?
The is no such way to schedule the steps of the Jenkins Job. But the Following approach may help you to achieve what you are trying to do.
Step-1: Install "Schedule Build Plugin" which will provide you a capability to schedule the job to run on certain data and time.
Step-2: Create a pipeline job with the code that you want to execute at a certain date and time. (Note: You will not schedule this job at this time)
Step-3: Remove the pipeline code that you already copied to another pipeline job and add a build step to schedule the build on the second pipeline that you will create in step-2.
I use Job DSL Plugin to generate my Jenkins builds. But sometimes I make small changes to the build in Jenkins and I want to port those changes back to my DSL script automatically. Is there any way to achieve this?
Currently there is no way to generate a Job DSL script for an existing job. This has been reported in the Jenkins issue tracker as JENKINS-16360 some time ago and someone even offered a bounty, but AFAIK no one is working on the issue.
Use case:
CI server polls some VSC repository and runs test suite for each revision. And if two or more revisions were commited, even in a relatively small time interval, I want the CI server to put each of them in queue, run tests for each, store the results, and never run tests again for those commits. And I don't want the CI server to launch jobs in parallel, to avoid performance issues and crashes in case of many simultaneous jobs.
Which CI server is able to handle this?
My additional, less important requirement is that I use Python and it is desirable to use software written in Python, so I looked at the Buildbot project, and I especially want to see reviews for this tool in the matter of is it usable in general and is it capable of replacing most popular solutions like Travis or Jenkins.
I have used jenkins to do this. (with subversion mainly, c/c++ build and also bash/python scripted jobs)
The easiest and default handling of VCS/SCM changes in jenkins is to poll for changes on a set time. A build is triggered if there is any change. More than one commit may be included in build (e.g. if 2 commits are done close together) when using this method. Jenkins shows links back to scm and scm update done as well as showing build logs and you can easily configure build outputs and test result presentation.
https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-Buildsbysourcechanges
What VCS/SCM are you using? Jenkins interfaces to a good few VCS/SCM:
https://wiki.jenkins-ci.org/display/JENKINS/Plugins#Plugins-Sourcecodemanagement
This question answers how to make Jenkins build on every subversion commit:
Jenkins CI: How to trigger builds on SVN commit
TeamCity is free (up to a number of builds and build agents) and feature-rich. It's very easy to install and configure, although it may take some time to find your way through the wealth of options. It is extremely well documented: http://www.jetbrains.com/teamcity/documentation/
It is written in Java but supports many tools natively and others through command-line execution, so you can build anything with it that you want. (I use it mostly for Ruby.) It understands the output of many testing tools; if you're not using one of them maybe yours can emulate their output. It's quite extensible; it has a REST API and a plugin API.
It can be configured to build on each commit, or to build all of the commits that arrived in a given time period, or to trigger in other ways. Docs here: http://confluence.jetbrains.com/display/TCD8/Configuring+VCS+Triggers
By default it starts a single build agent and runs one build at a time on that build agent. You can run more build agents for speed. If you don't want to run more than one build on a machine, only start one build agent on each machine.
I dont want that CI server would launch jobs in parallel to avoid
performance issues and crashes in cases of many simultanious jobs.
In buildbot you can limit the number of running jobs in a salve with max_build parameter or locks
As for Buildbot and Python, you may coordinate parallel builds by configuration, for example:
Modeling Parallel Processes: Steps
svn up
configure
make
make test
make dist
In addition, you can also try using a Triggerable scheduler for your builder which performs steps U,V,W.
From the docs:
The Triggerable scheduler waits to be triggered by a Trigger step (see
Triggering Schedulers) in another build. That step can optionally wait
for the scheduler's builds to complete. This provides two advantages
over Dependent schedulers.
References:
how to lock steps in buildbot
Coordinating Parallel Builds with
Buildbot
There is a Throttle Concurrent Builds Plugin for Jenkins and Hudson. It allows you to specify the number of concurrent builds per job. This is what it says on the plugin page:
It should be noted that Jenkins, by default, never executes the same Job in parallel, so you do not need to actually throttle anything if you go with the default. However, there is the option Execute concurrent builds if necessary, which allows for running the same Job multiple time in parallel, and of course if you use the categories below, you will also be able to restrict multiple Jobs.)
There is also Gitlab CI, a very nice modern Ruby project that uses runners to distribute builds so you could, I guess, limit the number of runners to 1 to get the effect you are after. It's tightly integrated with Gitlab so I don't know how hard it would be to use it as a standalone service.
www.gitlab.com
www.gitlab.com/gitlab-ci
To only run tests once for every revision you can do something like this:
build
post-build
check if the revision of the build is in /tmp/jenkins-test-run
if the revision is in the file skip tests
if the revision is NOT in the file run tests
if we ran the tests then write the ID in /tmp/jenkins-test-run
In Jenkins, If one build is currently running and next one is in pending state then what should i do so that running one should get aborted and next pending one should start running and so on.
I have to do it for few projects and each project has few jobs in it, I tried to save build_number as env variable in one text file (build_number.txt) and take that number to abort previous triggered build but making build_number.txt file for each job is not looking efficient and then I have to create many build_number files for each job for every project.
Can anyone please suggest me some better approach
Thanks
Based on the comments, if sending too many emails is the actual problem, you can use Poll SCM to poll once in 15 minutes or so, or even specify quiet time for a job. This will ensure that build is taken once in 15 minutes. Users should locally test before they commit. But if Jenkins itself is used for verifying the commits I don't see anything wrong in sending an email if build fails. After all, they are supposed to know that, no matter even if they fixed it in a later update intentionally or unintentionally.
But if you still want to abort a running job if there are updates, you can try the following. Lets call the job to be aborted as JOB A
Create another job that listens on updates same as that of the job that needs to be aborted
Add build step to execute groovy script
In the groovy script use Jenkins APIs to check if JOB A is running. If yes, again use APIs to abort the job.
Jenkins APIs are available here
So I have a dynamic number of jobs which all have only one build step.
at some ("random") point of its execution, each job run some application which couldn't have more than one instance at a given time.
In general I do want parallel run of the jobs. but I still need some synchronization when two or more jobs trying to run the above application at the same time.
I though about using the Locks and Latches plugin , but I can't see how this will help me in my situation.
Ideas will be more than welcomed!
Run the application from a separate job (let's call it APP_JOB) that you would invoke via Parametrized Trigger Plugin (as a build step, not as a post-build step) from your other jobs, with an option to wait for it enabled. By default Jenkins won't run parallel instances of APP_JOB.