How can I exclude specific days from a Jenkins scheduld?
For example: every 5 minutes from 7 to 17 and from monday to friday.
H/05 7-17 * * 1-5
But, if the day of the week is a public holiday, it should not run. How can I configure this?
Thx
Currently Jenkins crontab does not support a complicate logic such public holidays exclusion. However, there are some options out there you can use to accomplish that:
First
For my project I create my own holiday database, which it is a file containing the days I want to exclude eg.:
# /path/to/holidays
# New Year's Day
01-01-2017
# Christmas
12-25-2017
and I check it using a Jenkins shell script, as it was proposed here.
For example for the above file format:
#!/bin/bash
TODAY="`date +%m-%d-%Y`"
if grep -q $TODAY /path/to/holidays; then
echo Skipping holiday for $*
exit 0
fi
$*
Second
A more robust solution but more complicate is to create your own plugin based on the
Run Condition Example Plugin in which you exclude the public holidays of your Country such as this plugin.
There is another option now, with the Jenkins Working Hours Plugin.
Using the plugin, you can configure both working hours:
Or Holidays as 'Excluded Days':
The working hours plugin allows you to set up a schedule of allowable build times; projects can opt in to use the schedule to prevent them from running outside of configured allowable build times. If a build is scheduled during non-working hours then it is kept in the build queue until the next allowable time.
Jobs opt in via the enforceBuildSchedule job parameter, which is provided by this plugin. It can optionally take in a branches parameter to limit it's usage to only those branches. This only works in MultiBranchPipelines.
Usage
Sample job (scripted pipeline):
node {
properties([enforceBuildSchedule()])
stage('Do some stuff') {
echo 'this can wait til morning'
}
}
Sample job (declarative pipeline):
pipeline {
agent any
options {
enforceBuildSchedule()
}
stages {
stage('Do some stuff') {
steps {
echo 'this can wait til morning'
}
}
}
}
Sample job with branches parameter (works in both declarative and scripted):
node {
properties([enforceBuildSchedule(branches: ['dev', 'qa', 'prod')])
stage('Do some stuff') {
echo 'this can wait til morning'
}
}
Using the idea of one of the answers and using Declarative + Script block I created this:
stage('Check if Today is Holiday') {
steps {
// Based on idea from https://stackoverflow.com/a/41219757/7820857
script {
IS_HOLIDAY = sh(script: 'grep -q $(date +%Y-%m-%d) /etc/holidays', returnStatus: true)
if (IS_HOLIDAY == 0) {
currentBuild.result = 'ABORTED'
error ('Today is Holiday according to the file /etc/holidays inside the Jenkins server')
}
}
}
}
This will depend on the file /etc/holidays inside the Jenkins server. Adding this additional Stage before will help you to identify if the day mentioned is holiday and exit with error message, or not and continue with the rest of the stages.
I will like that the Working Hours Plugin worked for this but they queue the jobs in case that the day is inside the Excluded days, but I need to cancel the job execution. A feature request exist for that user case.
Related
We would like to run the Jenkins pipeline:
nightly, but only if there was any commit in the chosen branches (dev or master)
and after every commit
With the caveat that nightly build should also tigger additional step (long-running integration tests).
Is it possible? How can we configure this in Jenkinsfile?
So, your question boils down to several:
How can I run a nightly build?
How can I stop a build if no commits were made in the last 24 hours?
How can I run a "long-running" stage, but not always?
Let's address these one by one.
You can have a nightly build by using cron or parameterizedCron, if you install the ParameterizedCron plugin. As you seem to want to run your job in all the branches, this may look like e.g.
pipeline {
agent any
triggers {
// schedule a nightly build at random time after 0am on dev branch, 1am on master
cron(env.BRANCH_NAME == 'dev' ? '0 * * * *' : '')
cron(env.BRANCH_NAME == 'master' ? '1 * * * *' : '')
To address the other points, you need to know why your build is running. It may be triggered by a person, or cron, or a commit. Once you make sure it's cron that triggered the build, you may want to explore git log for the time of the latest commit.
def currentBuildReason() {
def timerCause = currentBuild.rawBuild.getCause(hudson.triggers.TimerTrigger.TimerTriggerCause)
if (timerCause) { echo "Build reason: Build was started by timer"; return "TIMER" }
timerCause = currentBuild.rawBuild.getCause(org.jenkinsci.plugins.parameterizedscheduler.ParameterizedTimerTriggerCause)
if (timerCause) { echo "Build reason: Build was started by parameterized timer"; return "TIMER" }
def userCause = currentBuild.rawBuild.getCause(hudson.model.Cause$UserIdCause)
if (userCause) { echo "Build reason: Build was started by user"; return "USER" }
println "here are the causes"
echo "${currentBuild.buildCauses}"
return "UNKNOWN"
}
and later:
if ( currentBuildReason() == "TIMER" and env.BRANCH == "master" ) {
def gitLog = sh script: "git log", returnStdout: true
// look for the date, and stop the build
Finally, you can run steps on condition. One way to do it is to define a parameter for your job, indicating whether the integration tests should run. This will allow a person to run them even if not nightly, when they select the box. E.g.
pipeline {
parameters {
booleanParam(name: 'RUN_LONG_TEST', defaultValue: false,
description: "Should long-running tests execute?")
}
and later
stage('Integration tests') {
agent { node { label "integration_test_slave" }}
when {
beforeAgent true
expression {params.RUN_LONG_TEST}
}
steps {
// run integration test
}
}
To schedule the nightly with the param selected, you need the parameterizedCron plugin:
pipeline {
agent any
triggers {
// schedule a nightly build at random time after 0am on dev branch, 1am on master
parameterizedCron(env.BRANCH_NAME == 'dev' ? '0 * * * * % RUN_LONG_TEST=true' : '')
parameterizedCron(env.BRANCH_NAME == 'master' ? '1 * * * * % RUN_LONG_TEST=true'' : '')
Alternatively, you may disregard the parameter if only the nightlies should run it, and analyze the value of currentBuildReason(). When it equals to TIMER, these tests should run.
I am trying to implement Machine learning in my jenkins pipeline.
For that I need output data of pipeline for each build.
Some parameters that i need are:
Which user triggered the pipeline
Duration of pipeline
Build number with its details
Pipeline pass/fail
If fail, at which stage it failed.
Error in the failed stage. (Why it failed)
Time required to execute each stage
Specific output of each stage (For. eg. : If a stage contains sonarcube execution then output be kind of percentage of codesmells or code coverage)
I need to fetch these details for all builds. How can get it?
There is jenkins api that can be implemented in python but i was able to get only JOB_NAME, Description of job, IS job Enabled.
These details werent useful.
There are 2 ways to get some of data from your list.
1. Jenkins API
For first 4 points from the list, you can use JSON REST API for a specific build to get those data. Example API endpoint:
https://[JENKINS_HOST]/job/[JOB_NAME]/[BUILD_NUMBER]/api/json?pretty=true
1. Which user triggered the pipeline
This will be under actions array in response, identyfi object in array by "_class": "hudson.model.CauseAction" and in it you will have shortDescription key which will have that information:
"actions": [
{
"_class": "hudson.model.CauseAction",
"causes": [
{
"_class": "hudson.triggers.SCMTrigger$SCMTriggerCause",
"shortDescription": "Started by an SCM change"
}
]
},
2. Duration of pipeline
It can be found under key: "duration". Example
"duration": 244736,
3. Build number with its details
I don't know what details you need, but for build number look for "number" key:
"number": 107,
4. Pipeline pass/fail
"result": "SUCCESS",
If you need to extract this information for all builds, run GET request for job API https://[JENKINS_HOST]/job/[JOB_NAME]/api/json?pretty=trueand extract all builds, then run above-mentioned request per build you have extracted.
I will write later a dummy python script to do just that.
2. Dump data in Jenkinsfile
There is also a possibility to dump some that information from Jenkinfile in post action.
pipeline {
agent any
stages {
stage('stage 1') {
steps {
sh 'echo "Stage 1 time: ${YOUR_TIME_VAR}" > job_data.txt'
}
}
}
post {
always {
sh 'echo "Result: ${result}" > job_data.txt'
sh 'echo "Job name: ${displayName}" > job_data.txt'
sh 'echo "Build number: ${number}" > job_data.txt'
sh 'echo "Duration: ${duration}" > job_data.txt'
archiveArtifacts artifacts: 'job_data.txt', onlyIfSuccessful: false
}
}
}
List of available global variables for pipeline job can be found:
https://[JENKINS_HOST]/pipeline-syntax/globals#env
For rest, you will need to implement your own logic in Jenkinsfile.
Ad. 5
Create a variable which holds information about current stage. At the beginning of each stage change its value to the ongoing stage. At the end dump to file like rest variables. If pipeline will fail let's say on stage foo in post action this variable will have exact same value because if pipeline fails it won't go to next stage.
Ad. 6
I'm not sure what you want, a traceback, error code?
I guess you will probably need to implement your own logging function.
Ad. 7
Make a function for measuring time for each stage and dump value at the end.
Ad. 8
Also not sure what you mean. Like, build artifacts?
At the end of each build this file job_data.txt will be archived as build artifact which can be later downloaded.
If i will find more elegant and simple solution I'll edit this post.
Hope it helps in any way
EDIT 1
Here is the script I've mentioned earlier.
import requests
username = "USERNAME"
password = "PASSWORD"
jenkins_host = "JENKINS_HOST"
jenkins_job = "JOBNAME"
request_url = "{0:s}/job/{1:s}/api/json".format(
jenkins_host,
jenkins_job,
)
job_data = requests.get(request_url, auth=(username, password)).json()
builds = []
for build in job_data.get('builds'):
builds.append(build.get('number'))
for build in builds:
build_url = "{0:s}/job/{1:s}/{2:d}/api/json".format(
jenkins_host,
jenkins_job,
build,
)
build_data = requests.get(build_url, auth=(username, password)).json()
build_name = build_data.get('fullDisplayName')
build_number = build_data.get('number')
build_status = build_data.get('result')
build_duration = build_data.get('duration')
for action in build_data.get('actions'):
if action.get("_class") == "hudson.model.CauseAction":
build_trigger = action.get('causes')
print(build_name)
print(build_status)
print(build_duration)
print(build_number)
print(build_trigger)
Please note you might need to authorize with API Token depending on your security settings.
How to define post build actions for Jenkins multi pipeline project?
There is a separate option available when you have a simple project but not for multipipeline.
To add post build steps to a Multibranch Pipeline, you need to code these steps into the finally block, an example is below:
node {
try {
stage("Checkout") {
// checkout scm
}
stage("Build & test") {
// build & Unit test
}
} catch (e) {
// fail the build if an exception is thrown
currentBuild.result = "FAILED"
throw e
} finally {
// Post build steps here
/* Success or failure, always run post build steps */
// send email
// publish test results etc etc
}
}
For most of the post-build steps you would want there are online examples of them on how to write in pipeline format. If you have any specific one please list it here
When you write a pipeline, you describe the whole flow yourself, which gives you great flexibility to do whatever you want, including running post-build steps.
You can see an example of using post-build steps in a pipeline I wrote:
https://github.com/geek-kb/Android_Pipeline/blob/master/Jenkinsfile
Example from that code:
run_in_stage('Post steps', {
sh """
# Add libCore.so files to symbols.zip
find ${cwd}/Product-CoreSDK/obj/local -name libCore.so | zip -r ${cwd}/Product/build/outputs/symbols.zip -#
# Remove unaligned apk's
rm -f ${cwd}/Product/build/outputs/apk/*-unaligned.apk
"""
})
I'm new to Jenkins Pipeline jobs, and I'm facing an issue I cannot solve.
I have a stage with a hardcoded sleep seconds value:
stage ("wait_prior_starting_smoke_testing") {
echo 'Waiting 5 minutes for deployment to complete prior starting smoke testing'
sleep 300 // seconds
}
But I would like to provide the time argument via a job (string) parameter SLEEP_TIME_IN_SECONDS. But whatever I have tried, I am not able to get it to work.
How do you convert a string parameter to the int time argument?
small improve for this page:
You also can use sleep(time:3,unit:"SECONDS") if you are interested in specifying time unit of your sleep
https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#sleep-sleep
Finally I did found a way to get this work:
stage ("wait_prior_starting_smoke_testing") {
def time = params.SLEEP_TIME_IN_SECONDS
echo "Waiting ${SLEEP_TIME_IN_SECONDS} seconds for deployment to complete prior starting smoke testing"
sleep time.toInteger() // seconds
}
Try this. It work for me.
stage ("wait_for_testing")
{
sh 'sleep 300'
}
With "sh" you start virtual terminal with shell. If you need write other command in one terminal, write ; after command.
For example:
sh 'pwd; sleep 300; echo "Hello World"'
I would like to have a Jenkins build flow that looks like this.
After the build is triggered all slaves run the same job in parallel (a setup job).
If any slaves fail this job they should not continue on.
For the all the slaves that to pass that job, they should grab a job out of a pool of jobs that need to be completed. And once a slave completes a job they should go back to complete another job in the pool.
I have only started working with Jenkins a few weeks ago and they way I have it setup now is as each job is picked up by a slave they have to run the setup job first. This really slows down build times because I have about 30 jobs and the setup takes ~2 minutes.
I am using Jenkins as an automated testing platform and all the jobs in the job pool can run independently of each other. I have 5 slaves currently and ~30 jobs.
The following should do the trick:
def jobPool = new ArrayDeque()
jobPool.add({
echo "Doing stuff on ${env.NODE_NAME}"
});
jobPool.add({
echo "Doing other stuff on ${env.NODE_NAME}, a little slower"
sleep 4
});
jobPool.add({
echo "Doing more stuff on ${env.NODE_NAME}, even slower"
sleep 10
});
jobPool.add({
echo "Doing stuff quick on ${env.NODE_NAME}"
});
jobPool.add({
echo "Doing stuff quicker on ${env.NODE_NAME}"
});
def par = [:]
for (x in ["master", "urban"]) {
def nodeName = x; // needed due to variable scoping
par[nodeName] = {
node (nodeName) {
try {
echo "Doing setup on ${env.NODE_NAME}!"
// Do you're setup
echo "Done with setup"
} catch (Exception e) {
echo "Will not use this node as it failed setup!"
return;
}
while (true) {
// echo "${jobPool.size()}"
def subTask = jobPool.poll()
//echo "${jobPool.size()} ${subTask}"
if (subTask == null) {
break;
}
// Might wan't try catch around the next line if you wan't to continue if a job fails
subTask()
}
}
}
}
parallel par
if (!jobPool.isEmpty()) {
error "Not all tasks was done!"
}
Simply add your "job pool jobs" to the jobPool variable and modify the setup part.
It seems like you want separate stages in the same job. This is made much easier in jenkins 2's pipelines. There are some pictures here:
https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Stage+View+Plugin
the [groovy] code ends up looking like this:
node {
stage 'Checkout'
svn 'https://svn.mycorp/trunk/'
stage 'Build'
sh 'make all'
stage 'Test'
sh 'make test'
}