Find Jenkin jobs running more than 24 hours - jenkins

I would like to find the Jenkin jobs in the master and client running more than 5 hours and kill it.
I tried multiple options using Thread.getAllStackTraces() and list all the jobs it was not helpful. Your help is much appreciated.

Use timeout
pipeline {
options {
timeout(time: 5, unit: 'HOURS')
}
stages {
steps {
sh '...'
}
}
}

Get all Jenkins jobs and their builds. Go through the list and check if there are builds running. If you find a build that is running, you can retrieve its time/timestamp/duration. Based on that time you could determine does that build is already running longer than X hours.
jenkins.model.Jenkins.instance.getAllItems(AbstractItem.class). each {
println "Job: " + it.fullName + " - " + it.class + " is running: " + it.isBuilding()
if(it.isBuilding()) {
def builds = it.getBuilds().each {
if(it.isBuilding()) {
println it.getTime() //getTimestamp(), getTimeInMillis(), getDuration() ...
}
}
}
}
WorkflowJob Docu
WorkflowRun Docu
Jenkins Core API

Related

How to block upstream/downstream build in Jenkins declarative pipeline?

I have 3 downstream build jobs which are triggered when 1 upstream job 'Project U' has been built successfully. Example:
triggers {
pollSCM('H/5 * * * *')
upstream(upstreamProjects: 'Project U', threshold: hudson.model.Result.SUCCESS)
}
This works as expected, however, if code changes are committed to all parts at the same time, the upstream and downstream builds start building simultaneously.
I want to avoid this, because the downstream builds will run twice, and the first run is quite useless as the upstream commit has not been built yet. So I would like to configure the downstream jobs to block their build while the upstream job is building.
I know how to do this in Jenkins Freestyle job in the user interface (also see this answer):
But I cannot find how to do this in a Jenkins declarative pipeline?
This approach works:
waitUntil {
def job = Jenkins.instance.getItemByFullName("Project U")
!job.isBuilding() && !job.isInQueue()
}
When this downstream is started, it will check if the upstream job is either active or queued, and if so, will wait until its build has finished.
I haven't been able to find out how to programmatically access the current job's upstream job(s), so the job names need to be copied. (There is a method getBuildingUpstream(), which would be much more convenient, but I haven't found a way to obtain the current Project object from the Job instance.)
I finally ended up creating this function in the Jenkins shared library:
/*
vars/waitForJobs.groovy
Wait until none of the upstream jobs is building or being queued any more
Parameters:
upstreamProjects String with a comma separated list of Jenkins jobs to check
(use same format as in upstream trigger)
*/
def call( Map params) {
projects = params['upstreamProjects'].split(', *')
echo 'Checking the following upstream projects:'
if ( projects.size() == 0) {
echo 'none'
} else {
projects.each {project ->
echo "${project}"
}
}
waitUntil {
def running = false
projects.each {project ->
def job = Jenkins.instance.getItemByFullName(project)
if (job == null) {
error "Project '${project} not found"
}
if (job.isBuilding()) {
echo "Waiting for ${project} (executing)"
running = true
}
if (job.isInQueue()) {
echo "Waiting for ${project} (queued for execution)"
running = true
}
}
return !running
}
}
The nice thing is that I can just copy the parameter from the upstream trigger, because it uses the exact same format. Here's an example of how it looks like:
pipeline {
[...]
triggers {
pollSCM('H/5 * * * *')
upstream(upstreamProjects: 'Project U1, Project U2, Project U3', threshold: hudson.model.Result.SUCCESS)
}
[...]
stages {
stage('Wait') {
steps {
script{
// Wait for upstream jobs to finish
waitForJobs(upstreamProjects: 'Project U1, Project U2, Project U3')
}
}
[...]
}
}
}

Jenkins pipeline script - stop previous build, and print message in failed build

So, I start my pipeline script with this piece of code:
node {
if (env.JOB_NAME != 'Company/project/develop' && env.JOB_NAME != 'Company/project/master'){
def jobName = env.JOB_NAME
def job = Jenkins.instance.getItemByFullName(jobName)
for (build in job.builds) {
if (build.isBuilding() && build != job.builds.first()) {
println '*' * 30
println 'All previous builds for this job have been aborted!'
println '*' * 30
build.doStop()
}}
}
}
And it works perfectly. It stops all previous builds for the same job name. The point of this is to optimize Jenkins when developers are pushing too much code, but latest push is only important one. Master and develop branches are excluded from this rule.
I would like to expand this a little bit, by adding interruption cause. I would like to print into stopped build for example: "This job has been stop because is obsolete..."
I have tried a lot of things posted on stackoverflow, but I didn't succeed.
Thanks
Here is my implementation of similar function. It will abort previous running build, and show cause "Aborted by newer build #123"
import hudson.model.Result
import hudson.model.Run
import jenkins.model.CauseOfInterruption.UserInterruption
Run previousBuild = currentBuild.rawBuild.getPreviousBuildInProgress()
while (previousBuild != null) {
if (previousBuild.isInProgress()) {
def executor = previousBuild.getExecutor()
if (executor != null) {
echo ">> Aborting older build #${previousBuild.number}"
executor.interrupt(Result.ABORTED, new UserInterruption("Aborted by newer build #${currentBuild.number}"))
}
}
previousBuild = previousBuild.getPreviousBuildInProgress()
}

How to get an id of build from which current build has restarted from stage?

I've a Jenkins pipeline and it has 5 stages. Suppose I run it and it has build id 5, but it fails at 3rd stage.
Now I re-run build 5 using Restart from failed stage feature and current build is with id 7(as meanwhile someone ran with id 6). NOw in current running build with id 7, I want to get id(which is 5) of build from which this build has restarted.
Is there any api using which I can get an id of build from which current build has restarted?
You can get this information through currentBuild.rawBuild.getCause(RestartDeclarativePipelineCause):
Working pipeline example:
pipeline{
agent any
stages {
stage('Stage 1') {
steps {
echoRestartedInfo()
}
}
stage('Stage 2') {
steps {
echoRestartedInfo()
}
}
}
}
void echoRestartedInfo() {
def restartCause = currentBuild.rawBuild.getCause(
org.jenkinsci.plugins.pipeline.modeldefinition.causes.RestartDeclarativePipelineCause )
if( restartCause ) {
def originRunNumber = restartCause.originRunNumber
def originStage = restartCause.originStage
echo "Restarted from build $originRunNumber, stage '$originStage'"
}
else {
echo "Normal run"
}
}
Because of using rawBuild this doesn't work in sandbox. Move the code into a shared library to work around this restriction.

jenkins complicated buildflow, is it possible?

I would like to have a Jenkins build flow that looks like this.
After the build is triggered all slaves run the same job in parallel (a setup job).
If any slaves fail this job they should not continue on.
For the all the slaves that to pass that job, they should grab a job out of a pool of jobs that need to be completed. And once a slave completes a job they should go back to complete another job in the pool.
I have only started working with Jenkins a few weeks ago and they way I have it setup now is as each job is picked up by a slave they have to run the setup job first. This really slows down build times because I have about 30 jobs and the setup takes ~2 minutes.
I am using Jenkins as an automated testing platform and all the jobs in the job pool can run independently of each other. I have 5 slaves currently and ~30 jobs.
The following should do the trick:
def jobPool = new ArrayDeque()
jobPool.add({
echo "Doing stuff on ${env.NODE_NAME}"
});
jobPool.add({
echo "Doing other stuff on ${env.NODE_NAME}, a little slower"
sleep 4
});
jobPool.add({
echo "Doing more stuff on ${env.NODE_NAME}, even slower"
sleep 10
});
jobPool.add({
echo "Doing stuff quick on ${env.NODE_NAME}"
});
jobPool.add({
echo "Doing stuff quicker on ${env.NODE_NAME}"
});
def par = [:]
for (x in ["master", "urban"]) {
def nodeName = x; // needed due to variable scoping
par[nodeName] = {
node (nodeName) {
try {
echo "Doing setup on ${env.NODE_NAME}!"
// Do you're setup
echo "Done with setup"
} catch (Exception e) {
echo "Will not use this node as it failed setup!"
return;
}
while (true) {
// echo "${jobPool.size()}"
def subTask = jobPool.poll()
//echo "${jobPool.size()} ${subTask}"
if (subTask == null) {
break;
}
// Might wan't try catch around the next line if you wan't to continue if a job fails
subTask()
}
}
}
}
parallel par
if (!jobPool.isEmpty()) {
error "Not all tasks was done!"
}
Simply add your "job pool jobs" to the jobPool variable and modify the setup part.
It seems like you want separate stages in the same job. This is made much easier in jenkins 2's pipelines. There are some pictures here:
https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Stage+View+Plugin
the [groovy] code ends up looking like this:
node {
stage 'Checkout'
svn 'https://svn.mycorp/trunk/'
stage 'Build'
sh 'make all'
stage 'Test'
sh 'make test'
}

Jenkins: remove old builds with command line

I delete old jenkins builds with rm where job is hosted:
my_job/builds/$ rm -rf [1-9]*
These old builds are still visible in job page.
How to remove them with command line?
(without the delete button in each build user interface)
Here is another option: delete the builds remotely with cURL. (Replace the beginning of the URLs with whatever you use to access Jenkins with your browser.)
$ curl -X POST http://jenkins-host.tld:8080/jenkins/job/myJob/[1-56]/doDeleteAll
The above deletes build #1 to #56 for job myJob.
If authentication is enabled on the Jenkins instance, a user name and API token must be provided like this:
$ curl -u userName:apiToken -X POST http://jenkins-host.tld:8080/jenkins/job/myJob/[1-56]/doDeleteAll
The API token must be fetched from the /me/configure page in Jenkins. Just click on the "Show API Token..." button to display both the user name and the API token.
Edit: As pointed out by yegeniy in a comment below, one might have to replace doDeleteAll by doDelete in the URLs above to make this work, depending on the configuration.
It looks like this has been added to the CLI, or is at least being worked on: http://jenkins.361315.n4.nabble.com/How-to-purge-old-builds-td385290.html
Syntax would be something like this: java -jar jenkins-cli.jar -s http://my.jenkins.host delete-builds myproject '1-7499' --username $user --password $password
Check your home jenkins directory:
"Manage Jenkins" ==> "Configure System"
Check field "Home directory" (usually it is /var/lib/jenkins)
Command for delete all jenkins job builds
/jenkins_home/jobs> rm -rf */builds/*
After delete should reload config:
"Manage Jenkins" ==> "Reload Configuration from Disk"
You can do it by Groovy Scripts using Hudson API.. Access your jenkins instalation
http://localhost:38080/script.
For Example, for deleting all old builds of all projects using the follow script:
Note: Take care if you use Finger Prints , you will lose all history.
import hudson.model.*
// For each project
for(item in Hudson.instance.items) {
// check that job is not building
if(!item.isBuilding()) {
System.out.println("Deleting all builds of job "+item.name)
for(build in item.getBuilds()){
build.delete()
}
}
else {
System.out.println("Skipping job "+item.name+", currently building")
}
}
Or for cleaning all workspaces :
import hudson.model.*
// For each project
for(item in Hudson.instance.items) {
// check that job is not building
if(!item.isBuilding()) {
println("Wiping out workspace of job "+item.name)
item.doDoWipeOutWorkspace()
}
else {
println("Skipping job "+item.name+", currently building")
}
}
There are a lot of examples on the Jenkins wiki
Is there a reason you need to do this manually instead of letting Jenkins delete old builds for you?
You can change your job configuration to automatically delete old builds, based either on number of days or number of builds. No more worrying about it or having to keep track, Jenkins just does it for you.
The following script cleans old builds of jobs. You should reload config from disk if you delete build manually:
import hudson.model.*
for(item in Hudson.instance.items) {
if (!item.isBuilding()) {
println("Deleting old builds of job " + item.name)
for (build in item.getBuilds()) {
//delete all except the last
if (build.getNumber() < item.getLastBuild().getNumber()) {
println "delete " + build
try {
build.delete()
} catch (Exception e) {
println e
}
}
}
} else {
println("Skipping job " + item.name + ", currently building")
}
}
From Script Console Run this, but you need to change the job name:
def jobName = "name"
def job = Jenkins.instance.getItem(jobName)
job.getBuilds().each { it.delete() }
job.nextBuildNumber = 1
job.save()
From Jenkins Scriptler console run the following Groovy script to delete all the builds of jobs listed under a view:
import jenkins.model.Jenkins
hudson.model.Hudson.instance.getView('<ViewName>').items.each() {
println it.fullDisplayName
def jobname = it.fullDisplayName
def item = hudson.model.Hudson.instance.getItem(jobname)
def build = item.getLastBuild()
if (item.getLastBuild() != null) {
Jenkins.instance.getItemByFullName(jobname).builds.findAll {
it.number <= build.getNumber()
}.each {
it.delete()
}
}
}
def jobName = "MY_JOB_NAME"
def job = Jenkins.instance.getItem(jobName)
job.getBuilds().findAll { it.number < 10 }.each { it.delete() }
if you had 12 builds this would clear out builds 0-9 and you'd have 12,11,10 remaining. Just drop in the script console
This script will configure the build retention settings of all of the Jenkins jobs.
Change the values from 30 and 200 to suite you needs, run the script, then restart the Jenkins service.
#!/bin/bash
cd $HOME
for xml in $(find jobs -name config.xml)
do
sed -i 's#<daysToKeep>.*#<daysToKeep>30</daysToKeep>#' $xml
sed -i 's#<numToKeep>.*#<numToKeep>200</numToKeep>#' $xml
done
The script below works well with Folders and Multibranch Pipelines. It preserves only 10 last builds for each job. That could be adjusted or removed (proper if) if needed. Run that from web script console (example URL: https://jenkins.company.com/script)
def jobs = Hudson.instance.getAllItems(hudson.model.Job.class)
for (job in jobs){
println(job)
def recent = job.builds.limit(10)
for(build in job.builds){
if(!recent.contains(build)){
println("\t Deleting build: " + build)
build.delete()
}
}
}
From my opinion all those answers are not sufficient, you have to do:
echo "Cleaning:"
echo "${params.PL_JOB_NAME}"
echo "${params.PL_BUILD_NUMBER}"
build_number = params.PL_BUILD_NUMBER as Integer
sleep time: 5, unit: 'SECONDS'
wfjob = Jenkins.instance.getItemByFullName(params.PL_JOB_NAME)
wfjob.getBuilds().findAll { it.number >= build_number }.each { it.delete() }
wfjob.save()
wfjob.nextBuildNumber = build_number
wfjob.save()
wfjob.updateNextBuildNumber(build_number)
wfjob.save()
wfjob.doReload()
Or the job will not be correctly reset and you have to hit build until you reach next free number in the meanwhile the jenkins log will show:
java.lang.IllegalStateException: JENKINS-23152: ****/<BUILD_NUMBER> already existed;

Resources