I have a working pipeline as below however it's not working over the stages. Is there any solution to implement it into graphical mode I mean something like stage/steps in the pipeline?
def nodes = ['node1','node2', 'node3']
for (int i = 0; i < nodes.size(); i++) {
step1(nodes[i])
step2(nodes[i])
step3(nodes[i])
}
def step1(node) {
echo 'in function, calling job on node ' + node
}
def step2(node) {
echo 'in function, calling job on node ' + node
}
def step3(node) {
echo 'in function, calling job on node ' + node
}
This is a very basic functionality question.
Rather than repeating the information in existing resources like one woudl expect on SO, I'll just provide some links
Jenkins tutorial about how to achieve this: https://www.jenkins.io/doc/book/pipeline/#stage
Pipeline Step reference for "Stage" dsl command: https://www.jenkins.io/doc/pipeline/steps/pipeline-stage-step/
Please clarify your question after reading thru this material, if need be. Thanks.
Related
I have a Jenkins pipeline that I'd like to run on either a params-specified agent or master. The pipeline code that implements this is:
pipeline {
agent { label "${params.agent} || master" }
...
}
I've surmised, from the following posts, that the || operator needs to be inside the (double-)quotes:
Can I define multiple agent labels in a declarative Jenkins Pipeline?
https://serverfault.com/questions/1074089/how-to-apply-multiple-labels-to-jenkins-nodes
Jenkinsfile - agent matching multiple labels
When I run this job, it seems to always run on master.
When I switch the order of ${params.agent} and master in the agent statement, it seems to still always run on master.
If I remove " || master" from the agent statement, then the job runs on the params-specified agent.
Question: Is my observation that Jenkins "prefers" master a coincidence, or is there something wrong with the syntax that's making Jenkins default to master?
Is there some way to have Jenkins prefer not-master so that I can test validity of the agent statement?
So, when Jenkins encounters the line
agent { label "${params.agent} || master" }
it will do exactly one of the following:
schedule your job on one of the nodes that match that label; or
get stuck until there's a node that matches that label, or until aborted.
With regards to option 1, there's no guarantee that it will do a round-robin, a random choice, or prefer some nodes but not the others, etc. In practice, when several nodes match, Jenkins will prefer the node that ran your pipeline in the past. This is a reasonable behavior — if there's a workspace already on that node, some operations (like git checkout) may happen faster, saving time.
With regards to option 2, that's also a reasonable behavior. We actually use that to schedule a job on a non-existing label, while manipulating the labels to produce one that would match.
So, there's nothing wrong with your syntax, and Jenkins is behaving as designed.
If you want to implement some custom rule — like "always try a different node", or "try to use master as little as possible" — you have to code that.
Example pipeline (note I haven't checked it):
import hudson.model.Hudson
properties([
parameters([
string(name: 'DEPLOY_ON', defaultValue: 'node_name',
description: 'try to run on this node, or master'),
])
])
resulting_node_name = ''
pipeline {
agent { node { label 'master' } }
stages {
stage ('Do on master') {
steps {
script {
resulting_node_name = params.DEPLOY_ON
// note: this gets node by name, but you can get by label if you wish
def slave = Jenkins.instance.getNode(resulting_node_name)
if (slave == null) {
currentBuild.result = 'FAILURE'
}
def computer = slave.computer
if (computer == null || computer.getChannel() == null || slave.name != params.DEPLOY_ON) {
println "Something wrong with the slave object, setting master"
resulting_node_name = 'master'
}
printSlaveInfo(slave)
computer = null
slave = null
}
}
}
stage('Do on actual node') {
agent { node { label resulting_node_name } }
steps {
script {
println "Running on ${env.NODE_NAME}"
}
}
}
}
}
#NonCPS
def printSlaveInfo(slave) {
// some info that you can use to choose the least-busy, best-equipped, etc.
println('====================')
println('Name: ' + slave.name)
println('getLabelString: ' + slave.getLabelString())
println('getNumExectutors: ' + slave.getNumExecutors())
println('getRemoteFS: ' + slave.getRemoteFS())
println('getMode: ' + slave.getMode())
println('getRootPath: ' + slave.getRootPath())
println('getDescriptor: ' + slave.getDescriptor())
println('getComputer: ' + slave.getComputer())
def computer = slave.computer
println('\tcomputer.isAcceptingTasks: ' + computer.isAcceptingTasks())
println('\tcomputer.isLaunchSupported: ' + computer.isLaunchSupported())
println('\tcomputer.getConnectTime: ' + computer.getConnectTime())
println('\tcomputer.getDemandStartMilliseconds: ' + computer.getDemandStartMilliseconds())
println('\tcomputer.isOffline: ' + computer.isOffline())
println('\tcomputer.countBusy: ' + computer.countBusy())
println('\tcomputer.getLog: ' + computer.getLog())
println('\tcomputer.getBuilds: ' + computer.getBuilds())
println('====================')
}
h pipeline/job has a trusted agent list, more job succeed on the agent, the agent sit in the top on the list, pipeline/agent will pick top agent from list.
If your pipeline already run on master for several times and all succeed, even you give another agent to choose, pipeline always pick the most trusted agent firstly
I came across this message with the code below
in JENKINS-44085.
If I already have a map of branches that contains 50 items, but I want to parallel them 5 at a time, how do I need to modify this code?
My code already has a map of 50 items in a var named branches.
// put a number of items into the queue to allow that number of branches to run
for (int i=0;i<MAX_CONCURRENT;i++) {
latch.offer("$i")
}
for (int i=0; i < 500; i++) {
def name = "$i"
branches[name] = {
def thing = null
// this will not allow proceeding until there is something in the queue.
waitUntil {
thing = latch.pollFirst();
return thing != null;
}
try {
echo "Hello from $name"
sleep time: 5, unit: 'SECONDS'
echo "Goodbye from $name"
}
finally {
// put something back into the queue to allow others to proceed
latch.offer(thing)
}
}
}
timestamps {
parallel branches
}
This question is a bit old, but for me the problem was also relevant yesterday. In some cases your Jenkins jobs may be light on Jenkins but high on some other system, so you want to limit it for that system. In my opinion using max executors per build agent is not the right way to do that because if your Jenkins cluster scales you will have to adjust stuff.
To answer your question, you probably want to do something like this:
Create a branches map with numeric indexes "0", "1", etc.
In the try block of that code you pasted have something like: build(my_branches[name])
At least that's how I was using that same workaround before. But then someone at work pointed out a better solution. I also commented this simpler solution in the Jira ticket you refered to. It requires the Lockable Resources Plugin: https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin
Go to: http://<your Jenkins URL>/configure and add X lockable resources with label "XYZ".
Use in your code as such:
def tests = [:]
for (...) {
def test_num="$i"
tests["$test_num"] = {
lock(label: "XYZ", quantity: 1, variable: "LOCKED") {
println "Locked resource: ${env.LOCKED}"
build(job: jobName, wait: true, parameters: parameters)
}
}
}
parallel tests
The nice thing about this is that you can use this across different jobs. In our case different jobs have a load on XYZ so having these global locks are very handy.
Is it possible to limit a job's build instance to run on up to x number of nodes?
For example, let's say I have 20 slaves and a parent job called "ParentJob". "ParentJob" is configured to run 20 child jobs concurrently, called "ChildJob", on any available slave. Since I need to kick off multiple ParentJobs, I want to limit the child jobs to 5 nodes per ParentJob build instance.
Basically, I want to kick off ParentJob #1, #2, and #3 concurrently, but I don't want all the ChildJobs spawned from ParentJob #1 to hog all the slaves. ChildJobs from ParentJob #1 should use nodes 1-5, ChildJobs from ParentJob #2 should use nodes 6-10, and ChildJobs from ParentJob #3 should use nodes 11-15.
I've looked at Throttle Concurrent Builds plugin and the Lockable Resource plugin, but they don't seem to address what I want to accomplish. They seem to work at the job level only, and not at the build level.
My other option is to write Groovy code to grab x available nodes, label them uniquely, and run the child jobs on these nodes. When the job is finished, I can clear the labels so they're available for the other builds.
Is there a simpler option or plugin that can do this? Maybe I'm over complicating this.
Hope that wasn't too confusing. Thanks in advance.
Since I couldn't find anything, I wrote the following Groovy script for my pipeline job.
This script will grab the first 3 slaves without a "TESTING_.*" label and append a new label. The jobs are then executed on those nodes only, without taking up all the slaves in my farm. Once the jobs are finished, the labels are removed.
node ("master") {
def new_label = "TESTING_" + org.apache.commons.lang.RandomStringUtils.random(7, true, true)
echo "New label for slaves: " + new_label
try {
stage('Reserve slaves') {
reserve_slaves(3, new_label)
}
stage('Smoke tests') {
// Do your parallel jobs here
// Pass in new_label to the job
}
}
finally {
stage('Return slaves') {
remove_label_from_slaves(new_label)
}
}
}
def reserve_slaves(number_of_slaves, new_label) {
def label_prefix = "TESTING_"
def slaves_with_new_label = 0
while (slaves_with_new_label < number_of_slaves) {
for (slave in jenkins.model.Jenkins.instance.slaves) {
def current_labels = slave.getLabelString()
if (!current_labels.contains(label_prefix)) {
echo "Adding label '${new_label}' to " + slave.name + ". Existing labels are: ${current_labels}"
slave.setLabelString(current_labels + " " + new_label)
slaves_with_new_label += 1
}
if (slaves_with_new_label >= number_of_slaves) {
break
}
}
if (slaves_with_new_label < number_of_slaves) {
echo "Waiting for nodes to become available..."
sleep(10) // in seconds
}
}
}
def remove_label_from_slaves(label) {
for (slave in jenkins.model.Jenkins.instance.slaves) {
def current_labels = slave.getLabelString()
if (current_labels.contains(label)) {
echo "Removing label '${label}' from " + slave.name + ". Existing labels are: ${current_labels}"
current_labels = current_labels.replace(label, '')
slave.setLabelString(current_labels)
}
}
}
I have a Jenkins Pipeline Job which has this code :
import hudson.model.*
import hudson.util.*
import hudson.scm.*
import hudson.scm.SubversionChangeLogSet.LogEntry
stage 'Build'
node('master'){
svn 'http://mysvn/url'
def build = Thread.currentThread()?.executable
def changeSet= build.getChangeSet()
.
.
}
The code is with unchecked 'sandbox' (as it presented on the picture).
and I get this error :
groovy.lang.MissingPropertyException: No such property: executable for class: java.lang.Thread
I am not familiar with the syntax for Thread.currentThread()?.executable
what does the '?' operator means.
I google it and found out about jenkins job-dsl plugin and didn't find anything about this operator.
I also tried the Script Console Plugin at : http://localhost:8080/script
and I fail for the same reason.
Does Pipeline Plugin support the Jenkins DSL-JOB ? should I import something in order to make it work ?
Here is the related ticket and the answer from cloudbees.. Ping back from there:
def changeLogSets = currentBuild.rawBuild.changeSets
for (int i = 0; i < changeLogSets.size(); i++) {
def entries = changeLogSets[i].items
for (int j = 0; j < entries.length; j++) {
def entry = entries[j]
echo "${entry.commitId} by ${entry.author} on ${new Date(entry.timestamp)}: ${entry.msg}"
def files = new ArrayList(entry.affectedFiles)
for (int k = 0; k < files.size(); k++) {
def file = files[k]
echo " ${file.editType.name} ${file.path}"
}
}
}
It's a groovy operator, '?' is to prevent NullpointerExceptions. It does the following only if the first is not null.
The Sandbox feature is to prevent certain calls so anybody can add scripts without admin approval, but very limited...
def build = Thread.currentThread()?.executable
Firstly, the above sentence can be explained like this,
Thread.currentThread() will get the current running thread, in normal case, it will be a instance of Jenkins Executor class, there is a attribute inside this class,
/**
* {#link hudson.model.Queue.Executable} being executed right now, or null if the executor is idle.
*/
#GuardedBy("lock")
private Queue.Executable executable;
Jenkins AbstractBuild implement this interface, so it means you will actually get AbstractBuild instance back.
However, this sentence is not work for pipeline related jobs since pipe line project have different structure compared with old Jenkins jobs. It don't extend AbstractBuild class.
This is why your script is not working.
About your requirement, since there is no AbstrctBuild class, so a lot of methods are actually cannot be used, like the one you used.
No idea if there is a smart way to get the changset inside pipeline job, or maybe you need restructure your job to adapt pipeline plugin.
Br,
Tim
I am trying to do a poc of jenkins pipeline as code. I am using the Github organization folder plugin to scan Github orgs and create jobs per branch. Is there a way to explicitly define the names for the pipeline jobs that get from Jenkinsfile? I also want to add some descriptions for the jobs.
You need to use currentBuild like below. The node part is important
node {
currentBuild.displayName = "$yournamevariable-$another"
currentBuild.description = "$yourdescriptionvariable-$another"
}
Edit: Above one renames build where as Original question is about renaming jobs.
Following script in pipeline will do that(this requires appropriate permissions)
item = Jenkins.instance.getItemByFullName("originalJobName")
item.setDescription("This description was changed by script")
item.save()
item.renameTo("newJobName")
I'm late to the party on this one, but this question forced me in the #jenkins chat where I spent most of my day today. I would like to thank #tang^ from that chat for helping solve this in a graceful way for my situation.
To set the JOB description and JOB display name for a child in a multi-branch DECLARATIVE pipeline use the following steps block in a stage:
steps {
script {
if(currentBuild.rawBuild.project.displayName != 'jobName') {
currentBuild.rawBuild.project.description = 'NEW JOB DESCRIPTION'
currentBuild.rawBuild.project.setDisplayName('NEW JOB DISPLAY NAME')
}
else {
echo 'Name change not required'
}
}
}
This will require that you approve the individual script calls through the Jenkins sandbox approval method, but it was far simpler than anything else I'd found across the web about renaming the actual children of the parent pipeline. The last thing to note is that this should work in a Jenkinsfile where you can use the environment variables to manipulate the job items being set.
I tried to used code snippet from accepted answer to describe my Jenkins pipeline in Jenkinsfile. I had to wrap code snippet into function with #NonCPS annotation and use def for item variable. I have placed code snippet in root of Jenkinsfile, not in node section.
#NonCPS
def setDescription() {
def item = Jenkins.instance.getItemByFullName(env.JOB_NAME)
item.setDescription("Some description.")
item.save()
}
setDescription()