We're considering using the Jenkins Pipeline plugin for a rather complex project consisting of several deliveries that need to be build using different tools (on different machines) before being merged. Still, it seems to be easy enough to do a complete build with a single Jenkinsfile, and I like the automatic discovery of git branches that comes with Pipeline.
However, at this point, we have jobs for each of the deliveries and use a build-flow based "meta" job to orchestrate the individual jobs. The nice thing about this is that it also allows starting just one individual job if only small changes were made, just to see whether this delivery still compiles.
To emulate this, some ideas came to mind:
Use different Jenkinsfiles for the deliveries and load them in the top-level Jenkinsfile; it seems that the Multibranch Pipeline job does not allow configuring the Jenkinsfile to use yet (https://issues.jenkins-ci.org/browse/JENKINS-35415), however, so creating the jobs for the individual deliveries is still open.
Provide a configuration option for the "top-level" job and have ifs for all deliveries in the Jenkinsfile to be able to select which should be build. This would mix different build types in one pipeline, though, and, at the very least, mess up the estimation of the build time.
Are those viable options, or is there a better one?
What you could do is to write a pipelining script that has has "if"-guards around the single stages, like this:
stage "s1"
if (theStage in ["s1","all"]) {
sleep 2
}
stage "s2"
if (theStage in ["s2", "all"]) {
sleep 2
}
stage "s3"
if (theStage in ["s3", "all"]) {
sleep 2
}
Then you can make a "main" job that uses this script and runs all stages at once by setting the parameter "theStage" to "all". This job will collect the statistics when all stages are run at once and give you useful estimation times.
Furthermore, you can make a "partial run" job that uses this script and that is parametrized with the stage that you want to run. The estimation will not be very useful, though.
Note that I put the stage itself to the main script and put only the execution code into the conditional, as suggested by Martin Ba. This makes sure that the visualization of the job is more reliable
As an expansion of the previous answer, I would propose something like that:
def stageIf(String name, Closure body) {
if (params.firstStage <= name && params.lastStage >= name) {
stage(name, body)
} else {
stage(name) {
echo "Stage skipped: $name"
}
}
}
node('linux') {
properties([
parameters([
choiceParam(
name: 'firstStage',
choices: '1.Build\n' +
'2.Docker\n' +
'3.Deploy',
description: 'First stage to start',
defaultValue: '1.Build',
),
choiceParam(
name: 'lastStage',
choices: '3.Deploy\n' +
'2.Docker\n' +
'1.Build',
description: 'Last stage to start',
defaultValue: '3.Deploy',
),
])
])
stageIf('1.Build') {
// ...
}
stageIf('3.Deploy') {
// ...
}
}
Not as perfect as I wish but at least its working.
Related
Can someone please let me know how i can trigger multiple Jenkins jobs? We have 5-6 jobs that we need to trigger manually post some technical upgrades. We are having to navigate to these jobs manually and click on 'Build'. Is there anyway i can create a new job or shell script that will help me in triggering all of these jobs with a single click/run.
Simplest approach will be creating a new pipeline which will trigger other pipelines.
You can also make stages parallel to faster execution if order is not issue.
You can find detailed answer and approach here
Yes, you can do it in both ways that you mentioned. Creating a new job that will run all these other jobs as downstream jobs is the fastest and easiest.
To do this, create a new job, and if you are using declarative Jenkins, then specify the jobs to be triggered:
stage ('trigger-multiple-jobs') {
parallel {
stage ('first job') {
steps {
build([
job : 'JobName',
wait : false,
parameters: [
string(name: 'PARAM_1', value: "${PARAM_1}")
]
])
}
}
stage ('second job') {
steps {
build([
job : 'JobName2',
wait : false
])
}
}
}
}
You can alternatively create a freestyle job, navigate down to the Trigger downstream jobs section, and set the jobs you want to be triggered from the drop down.
If you want to use a shell script, then you can trigger the jobs using API calls. This is described well in this answer.
I have Jenkins Pipeline which is triggering for different projects. However the only difference in all the pipelines is just the name.
So I have added a parameter ${project} in parameter of jenkins and assigned it a value of the name of the project.
We have a number of projects and I am trying to find a better way through which I can achieve this.
I am thinking how can we make the parameter run with different parameters for all the projects without actually creating different projects under jenkins.
I am pasting some screenshot for you to understand what exactly I want to achieve.
As mentioned here, this is a radioserver project, having a pipeline which has ${project} in it.
How can I give multiple values to that {project} from single jenkins job?
IF you have any doubts please message me or add a comment.
You can see those 2 projects I have created, it has all the contents same but just the parameterized value is different, I am thinking how can I give the different value to that parameter.
As you can see the 2 images is having their default value as radioserver, nrcuup. How can I combine them and make them run seemlessly ?
I hope this will help. Let me know if any changes required in answer.
You can use conditions in Jenkins. Based on the value of ${PROJECT}, you can then execute the particular stage.
Here is a simple example of a pipeline, where I have given choices to select the value of parameter PROJECT i.e. test1, test2 and test3.
So, whenever you select test1, jenkins job will execute the stages that are based on test1
Sample pipeline code
pipeline {
agent any
parameters {
choice(
choices: ['test1' , 'test2', 'test3'],
description: 'PROJECT NAME',
name: 'PROJECT')
}
stages {
stage ('PROJECT 1 RUN') {
when {
expression { params.PROJECT == 'test1' }
}
steps {
echo "Hello, test1"
}
}
stage ('PROJECT 2 RUN') {
when {
expression { params.PROJECT == 'test2' }
}
steps {
echo "Hello, test2"
}
}
}
}
Output:
when test1 is selected
when test2 is selected
Updated Answer
Yes, it is possible to trigger the job periodically with a specific parameter value using the Jenkins plugin Parameterized Scheduler
After you save the project with some parameters (like above mentioned pipeline code), go back again to the Configure and under Build Trigger, you can see the option of Build periodically with parameters
Example:
I will here run the job for PROJECT=test1 every even minutes and PROJECT=test2 every uneven minutes. So, below is the configuration
*/2 * * * * %PROJECT=test1
1-59/2 * * * * %PROJECT=test2
Please change the crontab values according to your need
Output:
I have Jenkins Pipeline jobs, where the only difference between the jobs is a parameter, a single "name" value, I could even use the multibranch job name (though not what it's passing as JOB_NAME which is the BRANCH name, sadly none of the envs look suitable without parsing). It would be great if I could set this outiside of the Jenkinsfile, since then I could reuse the same jenkinsfile for all the various jobs.
Add this to your Jenkinsfile:
properties([
parameters([
string(name: 'myParam', defaultValue: '')
])
])
Then, once the build has run once, you will see the "build with parameters" button on the job UI.
There you can input the parameter value you want.
In the pipeline script you can reference it with params.myParam
Basically you need to create a jenkins shared library example name myCoolLib and have a full declarative pipeline in one file under vars, let say you call the file myFancyPipeline.groovy.
Wanted to write my examples but actually I see the docs are quite nice, so I'll copy from there. First the myFancyPipeline.groovy
def call(int buildNumber) {
if (buildNumber % 2 == 0) {
pipeline {
agent any
stages {
stage('Even Stage') {
steps {
echo "The build number is even"
}
}
}
}
} else {
pipeline {
agent any
stages {
stage('Odd Stage') {
steps {
echo "The build number is odd"
}
}
}
}
}
}
and then aJenkinsfile that uses it (now has 2 lines)
#Library('myCoolLib') _
evenOrOdd(currentBuild.getNumber())
Obviously parameter here is of type int, but it can be any number of parameters of any type.
I use this approach and have one of the groovy scripts that has 3 parameters (2 Strings and an int) and have 15-20 Jenkinsfiles that use that script via shared library and it's perfect. Motivation is of course one of the most basic rules in any programming (not a quote but goes something like): If you have "same code" at 2 different places, something is not right.
There is an option This project is parameterized in your pipeline job configuration. Write variable name and a default value if you wish. In pipeline access this variable with env.variable_name
Original situation:
I have a job in Jenkins that is running an ant script. I easily managed to test this ant script on more then one software version using a "Multi-configuration project".
This type of project is really cool because it allows me to specify all the versions of the two software that I need (in my case Java and Matlab) an it will run my ant script with all the combinations of my parameters.
Those parameters are then used as string to be concatenated in the definition of the location of the executable to be used by my ant.
example: env.MATLAB_EXE=/usr/local/MATLAB/${MATLAB_VERSION}/bin/matlab
This is working perfectly but now I am migrating this scripts to a pipline version of it.
Pipeline migration:
I managed to implement the same script in a pipeline fashion using the Parametrized pipelines pluin. With this I achieve the point in which I can manually select which version of my software is going to be used if I trigger the build manually and I also found a way to execute this periodically selecting the parameter I want at each run.
This solution seems fairly working however is not really satisfying.
My multi-config project had some feature that this does not:
With more then one parameter I can set to interpolate them and execute each combination
The executions are clearly separated and in build history/build details is easy to recognize which settings hads been used
Just adding a new "possible" value to the parameter is going to spawn the desired executions
Request
So I wonder if there is a better solution to my problem that can satisfy also the point above.
Long story short: is there a way to implement a multi-configuration project in jenkins but using the pipeline technology?
I've seen this and similar questions asked a lot lately, so it seemed that it would be a fun exercise to work this out...
A matrix/multi-config job, visualized in code, would really just be a few nested for loops, one for each axis of parameters.
You could build something fairly simple with some hard coded for loops to loop over a few lists. Or you can get more complicated and do some recursive looping so you don't have to hard code the specific loops.
DISCLAIMER: I do ops much more than I write code. I am also very new to groovy, so this can probably be done more cleanly, and there are probably a lot of groovier things that could be done, but this gets the job done, anyway.
With a little work, this matrixBuilder could be wrapped up in a class so you could pass in a task closure and the axis list and get the task map back. Stick it in a shared library and use it anywhere. It should be pretty easy to add some of the other features from the multiconfiguration jobs, such as filters.
This attempt uses a recursive matrixBuilder function to work through any number of parameter axes and build all the combinations. Then it executes them in parallel (obviously depending on node availability).
/*
All the config axes are defined here
Add as many lists of axes in the axisList as you need.
All combinations will be built
*/
def axisList = [
["ubuntu","rhel","windows","osx"], //agents
["jdk6","jdk7","jdk8"], //tools
["banana","apple","orange","pineapple"] //fruit
]
def tasks = [:]
def comboBuilder
def comboEntry = []
def task = {
// builds and returns the task for each combination
/* Map the entries back to a more readable format
the index will correspond to the position of this axis in axisList[] */
def myAgent = it[0]
def myJdk = it[1]
def myFruit = it[2]
return {
// This is where the important work happens for each combination
node(myAgent) {
println "Executing combination ${it.join('-')}"
def javaHome = tool myJdk
println "Node=${env.NODE_NAME}"
println "Java=${javaHome}"
}
//We won't declare a specific agent this part
node {
println "fruit=${myFruit}"
}
}
}
/*
This is where the magic happens
recursively work through the axisList and build all combinations
*/
comboBuilder = { def axes, int level ->
for ( entry in axes[0] ) {
comboEntry[level] = entry
if (axes.size() > 1 ) {
comboBuilder(axes[1..-1], level + 1)
}
else {
tasks[comboEntry.join("-")] = task(comboEntry.collect())
}
}
}
stage ("Setup") {
node {
println "Initial Setup"
}
}
stage ("Setup Combinations") {
node {
comboBuilder(axisList, 0)
}
}
stage ("Multiconfiguration Parallel Tasks") {
//Run the tasks in parallel
parallel tasks
}
stage("The End") {
node {
echo "That's all folks"
}
}
You can see a more detailed flow of the job at http://localhost:8080/job/multi-configPipeline/[build]/flowGraphTable/ (available under the Pipeline Steps link on the build page.
EDIT:
You can move the stage down into the "task" creation and then see the details of each stage more clearly, but not in a neat matrix like the multi-config job.
...
return {
// This is where the important work happens for each combination
stage ("${it.join('-')}--build") {
node(myAgent) {
println "Executing combination ${it.join('-')}"
def javaHome = tool myJdk
println "Node=${env.NODE_NAME}"
println "Java=${javaHome}"
}
//Node irrelevant for this part
node {
println "fruit=${myFruit}"
}
}
}
...
Or you could wrap each node with their own stage for even more detail.
As I did this, I noticed a bug in my previous code (fixed above now). I was passing the comboEntry reference to the task. I should have sent a copy, because, while the names of the stages were correct, when it actually executed them, the values were, of course, all the last entry encountered. So I changed it to tasks[comboEntry.join("-")] = task(comboEntry.collect()).
I noticed that you can leave the original stage ("Multiconfiguration Parallel Tasks") {} around the execution of the parallel tasks. Technically now you have nested stages. I'm not sure how Jenkins is supposed to handle that, but it doesn't complain. However, the 'parent' stage timing is not inclusive of the parallel stages timing.
I also noticed is that when a new build starts to run, on the "Stage View" of the job, all the previous builds disappear, presumably because the stage names don't all match up. But after the build finishes running, they all match again and the old builds show up again.
And finally, Blue Ocean doesn't seem to vizualize this the same way. It doesn't recognize the "stages" in the parallel processes, only the enclosing stage (if it is present), or "Parallel" if it isn't. And then only shows the individual parallel processes, not the stages within.
Points 1 and 3 are not completely clear to me, but I suspect you just want to use “scripted” rather than “Declarative” Pipeline syntax, in which case you can make your job do whatever you like—anything permitted by matrix project axes and axis filters and much more, including parallel execution. Declarative syntax trades off syntactic simplicity (and friendliness to “round-trip” editing tools and “linters”) for flexibility.
Point 2 is about visualization of the result, rather than execution per se. While this is a complex topic, the usual concrete request which is not already supported by existing visualizations like Blue Ocean is to be able to see test results distinguished by axis combination. This is tracked by JENKINS-27395 and some related issues, with design in progress.
I am experimenting with Jenkins pipeline and milestones and cannot figure out why Jenkins is not cancelling the previous build when a new build crosses the milestone.
Example Jenkinsfile
pipeline {
agent any
parameters {
booleanParam(defaultValue: true, description: '', name: 'userFlag')
}
stages {
stage("foo") {
steps {
milestone(ordinal: 1, label: "BUILD_START_MILESTONE")
sh 'sleep 1000'
}
}
}
}
Triggering this pipeline twice does not cancel the 1st job
Try this:
/* This method should be added to your Jenkinsfile and called at the very beginning of the build*/
#NonCPS
def cancelPreviousBuilds() {
def jobName = env.JOB_NAME
def buildNumber = env.BUILD_NUMBER.toInteger()
/* Get job name */
def currentJob = Jenkins.instance.getItemByFullName(jobName)
/* Iterating over the builds for specific job */
for (def build : currentJob.builds) {
/* If there is a build that is currently running and it's not current build */
if (build.isBuilding() && build.number.toInteger() != buildNumber) {
/* Than stopping it */
build.doStop()
}
}
}
I don't think the behavior is "If I'm a newer build that crosses this milestone, then all older build that crossed this milestone will be cancelled"
The actual behavior of the milestone step is that when a more recent pipeline crosses it first, then it prevents older pipeline from crossing that milestone.
I have a simple work around with milestone plugin, according to the document:
Builds pass milestones in order (taking the build number as sorter field).
Older builds will not proceed (they are aborted) if a newer one already passed the milestone.
When a build passes a milestone, any older build that passed the previous milestone but not this one is aborted.
Once a build passes the milestone, it will never be aborted by a newer build that didn't pass the milestone yet.
you can try something like this:
pipeline {
agent any
stages {
stage('Stop Old Build') {
steps {
milestone label: '', ordinal: Integer.parseInt(env.BUILD_ID) - 1
milestone label: '', ordinal: Integer.parseInt(env.BUILD_ID)
}
}
}
}
you can put this at the start of any pipeline.
Assume you just start a new build, #5. The first milestone, will be used to passes #4's second milestone, and the second milestone(of #5) will be used to kill #4's process, if it's currently running.
The disableConcurrentBuilds property has been added to Pipeline. The Pipeline syntax snippet generator offers the following syntax hint:
properties([disableConcurrentBuilds(abortPrevious: true)])
That property is used on ci.jenkins.io to cancel older plugin build jobs when newer plugin build jobs start.
Declarative Pipeline also includes the disableConcurrentBuilds option that is documented in the Pipeline syntax page.
The declarative directive generator suggests the following:
options {
disableConcurrentBuilds abortPrevious: true
}
as per https://jenkins.io/blog/2016/10/16/stage-lock-milestone/, a pair of 'milestone()' works for me to kill the previous jobs while the pipeline kicked off for times,
stage('Build') {
// The first milestone step starts tracking concurrent build order
milestone()
node {
echo "Building"
}}
// The Deploy stage does not limit concurrency but requires manual input
// from a user. Several builds might reach this step waiting for input.
// When a user promotes a specific build all preceding builds are aborted,
// ensuring that the latest code is always deployed.
stage('Deploy') {
timeout(time: 60, unit: 'SECONDS') {input "Deploy?"}
milestone()
node {
echo "Deploying"
}
}
The last milestone helps kill previous builds if reached, say deploy button clicked for the above case. Or the locked resource released for the below case,
// This locked resource contains both Test stages as a single concurrency Unit.
// Only 1 concurrent build is allowed to utilize the test resources at a time.
// Newer builds are pulled off the queue first. When a build reaches the
// milestone at the end of the lock, all jobs started prior to the current
// build that are still waiting for the lock will be aborted
lock(resource: 'myResource', inversePrecedence: true){
node('test') {
stage('Unit Tests') {
echo "Unit Tests"
}
stage('System Tests') {
echo "System Tests"
}
}
milestone()
}
Building on #D.W.'s answer, i found a simple pattern that works. It seems to fit into D.W.'s bullet #3 (which is the official doc): When a build passes a milestone, Jenkins aborts older builds that passed the previous milestone but not this one.
Adding an earlier milestone that everything will pass, and then one after the thing that is going to wait, makes it all work like you think it should. In my case:
steps {
milestone 1
input 'ok'
milestone 2
}
Create two active builds with this, and only approve the second one. You'll see the first one get automatically canceled, because build 2 passed milestone 2 first.
Try taking out milestone 1, and you'll see that build 1 does not get canceled when build 2 passes milestone 2.
Adding the early milestone satisfies the requirement. It seems that a build has to pass any milestone before a future milestone passed by a newer build will cause it to cancel.