I'm trying to run a groovy script which is updating the build.description during the execution while running a process on a slave node.
My problem is that a "system groovy script' executes only on the master node and 'Jenkins groovy scripts' run on slave nodes but have no access to the build variable.
I have a script similar to this one:
import hudson.model.*
// works on slave node
def param = args[0]
// works on master node
//def param = build.getEnvironment(listener).get('Params')
def ws = new File(".").absolutePath
def myCommand = ws + "\\Something.exe " + param
def proc = myCommand.execute();
// Cannot use on slave
build.description = "Running executable..."
int exitVal = proc.waitFor();
// Cannot use on slave
build.description = "Executable finished"
Is there a way to modify the build variable on a "Jenkins Groovy Script" which runs on a slave?
Thanks in advance!
No.
Jenkins pipeline exposes more control on such things.
Following is from script we have. nodeexpression could a node name.
node(nodeexpression) {
println "env :"
echo sh(script: 'env|sort', returnStdout: true)
currentBuild.displayName = "branch ${BRANCH}:${MAIL_TO}"
currentBuild.description = "${BRANCH}:${MAIL_TO} : message -> ${MESSAGE}"
}
Related
I have a parameterized pipeline, accorindg to the user input, then some values are assigned to the variables, if you see the code there are 2 echo sections, one just after the switch statement, and another one within the pipeline steps in "preparation" stage, the first set of "echos" is displaying the correct values of the vars, while the second set of "echos" withing the pipeline steps is not, so it looks like the variables are not being received in the pipeline, how can I get this done?, (in the example below "Stress_Test" was seleted).
def local_path
def branch
def script_file
def datagen_file
switch (TEST_TYPE) {
case "Smoke_Test":
branch = "SmokeTest"
local_path = "/e/jmeter/apache-jmeter-5.3/bin/tcp-performance-engineering/${BRANCH}"
script_file = "PE_TCP_RESTAPI_June2020_SMK.jmx"
datagen_file= "TCP_JMeter_DataFiles_smk.yaml"
break
case "Regular_Load":
branch = "Regular"
local_path = "/e/jmeter/apache-jmeter-5.3/bin/tcp-performance-engineering/${branch}"
script_file = "PE_TCP_RESTAPI_July2020_Regular.jmx"
datagen_file= "TCP_JMeter_DataFiles_regular.yaml"
break
case "Peak_Load":
branch = "PeakTest"
local_path = "/e/jmeter/apache-jmeter-5.3/bin/tcp-performance-engineering/${branch}"
script_file = "PE_TCP_RESTAPI_July2020_Peak.jmx"
datagen_file= "TCP_JMeter_DataFiles_peak.yaml"
break
case "Stress_Test":
branch = "StressTest"
local_path = "/e/jmeter/apache-jmeter-5.3/bin/tcp-performance-engineering/${branch}"
script_file = "PE_TCP_RESTAPI_July2020_Stress.jmx"
datagen_file= "TCP_JMeter_DataFiles_stress.yaml"
break
default:
println "Test type was not set!"
break
}
echo "test type selected ${TEST_TYPE}"
echo "branch to checkout ${branch}"
echo "path in local ${local_path}"
echo "script name ${script_file}"
echo "datagen file name ${datagen_file}"
pipeline {
agent any
stages {
stage('Preparation...') {
steps{
sh '''
echo "test type selected ${TEST_TYPE}"
echo "branch to checkout ${branch}"
echo "path in local ${local_path}"
echo "script name ${script_file}"
echo 2datagen file name ${datagen_file}"
echo "****************************************"
*rest of the code is not relevant.
and this is what I'm getting in the output...
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] echo
test type selected Stress_Test
[Pipeline] echo
branch to checkout StressTest
[Pipeline] echo
path in local /e/jmeter/apache-jmeter-5.3/bin/tcp-performance-engineering/StressTest
[Pipeline] echo
script name PE_TCP_RESTAPI_July2020_Stress.jmx
[Pipeline] echo
datagen file name TCP_JMeter_DataFiles_stress.yaml
[Pipeline] node
Running on Jenkins in E:\jenkins\workspace\TCP_Performance_Test_V1
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Preparation...)
[Pipeline] sh
+ echo 'test type selected Stress_Test'
test type selected Stress_Test
+ echo 'branch to checkout '
branch to checkout
+ echo 'path in local '
path in local
+ echo 'script name '
script name
This is my first pipeline, any help will be really appreciated.
Jenkins pipelines are written in groovy and in groovy, ' is used for literal strings that do not support variable interpolation. In the question, you are using sh ''' ... ''' that makes everything within that block literal and executing as is. Meaning, all the variables are looked in the shell environment and not in the pipeline.
To fix, either change sh ''' ... ''' to sh """ ... """ or remove the sh block altogether.
Have a look at 4. Strings section of groovy documentation.
When I click "rebuild" from the page of a build jenkins rebuilds and runs a new job- a new job with a new jenkins build number.
How do I get the build number of the job where I executed the rebuild?
Im not talking about the previous build number.
Say Im on build 10. I go to build 5 and click rebuild. How do I that build number (5) from inside the pipeline like I can with env.BUILD_NUMBER?
I assume that you are using Groovy Pipeline and already know the Global Variable (see Global Variable Reference).
The currentBuild variable has a field rawBuild that return a hudson.model.Run object
Call rawBuildObject#getCauses() or rawBuildObject#getCauses() and return some Cause object.
script below:
node {
stage('test advance script') {
echo "current build number: ${currentBuild.number}"
echo "previous build number: ${currentBuild.previousBuild.getNumber()}"
def causes = currentBuild.rawBuild.getCauses()
echo "causes: ${causes}"
def rebuildCause0 = currentBuild.rawBuild.getCause(com.sonyericsson.rebuild.RebuildCause)
echo "rebuildCause0: ${rebuildCause0}"
echo "rebuild up number: ${rebuildCause0.getUpstreamBuild()}"
}
}
But as we discuss in chat, the Rebuilder Plugin use CauseAction in a wrong way. If it is fixed as this, console output should be:
current build number: 72
previous build number: 71
causes: [hudson.model.Cause$UserIdCause#679c1066, job/DMP/job/test-pipeline/63[hudson.model.Cause$UserIdCause#679c1066]]
rebuildCause0: job/DMP/job/test-pipeline/63[hudson.model.Cause$UserIdCause#679c1066]
rebuild up number: 63
Remember to scriptApproval when you see errors like this:
Scripts not permitted to use method hudson.model.Run
getCauses. Administrators can decide whether to approve or reject this
signature.
Assuming you can curl your own jenkins builds:
def getOriginalRebuildNum(String jobUrl = "${JOB_URL}"){
def urlJsonPath
def buildNumber
def result
ansiColor('xterm') {
try {
buildNumber = "${BUILD_NUMBER}"
urlJsonPath = "${JOB_URL}/${buildNumber}/api/json"
while (!buildNumber.isEmpty()) {
result = buildNumber
buildNumber = sh(
script:
"""
curl -s $urlJsonPath \
| jq -r '.actions[]
| select(."_class" == "hudson.model.CauseAction") .causes[]
| select(.upstreamBuild != null).upstreamBuild'
""",
returnStdout: true).trim()
urlJsonPath = "${JOB_URL}/${buildNumber}/api/json"
}
echo "Original Build: ${JOB_URL}/${result}/"
}
catch (err) {
"Error: Could not retrieve original Build Number from ${urlJsonPath} - $err"
result = "${BUILD_NUMBER}"
}
}
return result
}
I have a Jenkins declarative pipeline I have been running on the Jenkins master and it works fine. However, now that I have moved to trying to execute this on a slave node, the groovy scripts which are called in the pipeline can not access the files in the workspace.
My jenkinsfile looks like this...
pipeline {
agent {
label {
label "windows"
customWorkspace "WS-${env.BRANCH_NAME}"
}
}
stages {
stage('InitialSetup') {
steps {
"${env.WORKSPACE}/JenkinsScripts/myScript.groovy"
}
}
}
I can see on the slave that it is creating the workspace, doing the checkout from git, and executing the script correctly. However, if something in the script try's to interact with the files in the workspace it fails.
If I have something simple like this...
def updateFile(String filename) {
echo env.NODE_NAME
filename = "${env.WORKSPACE}/path/to/file"
def myFile = new File(filename)
<do other things with the file>
}
...it says it can not find the file specified. It gives me the path it is looking for and I can confirm the file exists, and that the code runs when just building on the master.
Why can the script not find the files this way when in can just running on the master node? I added the "echo env.NODE_NAME" command into my groovy file and it says the script is executing on the correct node.
Thanks.
Turns out Groovy File commands are considered insecure, and although they will run on the master, they will not run on the slave. If you call them from a script that has the agent set to another node, it will still execute the command just fine, just on the master node, not the agent. Here's an excerpt of an article post https://support.cloudbees.com/hc/en-us/articles/230922508-Pipeline-Files-manipulation
The operation with File class are run on master, so only works if build is run on master, in this example I create a file and check if I can access it on a node with method exists, it does not exist because the new File(file) is executed on master, to check this I search for folder Users that exist on my master but not in the node.
stage 'file move wrong way'
//it only works on master
node('slave') {
def ws = pwd()
def context = ws + "/testArtifact"
def file = ws + '/file'
sh 'touch ' + file
sh 'ls ' + ws
echo 'File on node : ' + new File(file).exists()
echo 'Users : ' + new File('/Users').exists()
sh 'mv ' + file + ' ' + context
sh 'ls ' + ws
}
To execute file manipulation command we recommend to use native commands.
This is a simple example of operations in shell
stage 'Create file'
sh 'touch test.txt'
stage 'download file'
def out='$(pwd)/download/maven.tgz'
sh 'mkdir -p ./download'
sh 'curl -L http://ftp.cixug.es/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz -o ' + out
stage 'move/rename'
def newName = 'mvn.tgz'
sh 'mkdir -p $(pwd)/other'
sh 'mv ' + out + ' ' + newName
sh 'cp ' + newName + ' ' + out
}
I run into this same issue recently. I had a python file that runs and writes the results to a JSON file. I was trying to access the JSON file to retrieve the data from there. Here is the code I was using inside a stage block of a declarative pipeline:
script {
def jsonSlurper = new JsonSlurper()
def fileParsed = new File("parameters.json")
def dataJSON = jsonSlurper.parse(fileParsed)
}
As everyone stated already, the above was failing with FileNotFoundException because anything inside script{} will only run on master and not the agent.
To work around the issue, I have used the Pipeline Utility Steps plugin (reference: https://plugins.jenkins.io/pipeline-utility-steps/ -- How to use: https://www.jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#writejson-write-json-to-a-file-in-the-workspace)
The plugin will allow you to do any read/write operation on multiple file formats.
Here is an example of the code I used after installing the plugin:
script {
def props = readJSON file: 'parameters.json'
println("just read it..")
println(props)
}
Note: I was using jenkins 2.249.1
I have implemented the code which automatically installs Groovy on slave (for scripted pipeline). Perhaps this solution is a little bit cumbersome, but pipelines don't offer any other way to achieve the same functionality as "Execute Groovy Script" stuff from the old Jenkins, because the plugin https://wiki.jenkins.io/display/JENKINS/Groovy+plugin is not supported yet for pipeline.
import hudson.tools.InstallSourceProperty;
import hudson.tools.ToolProperty;
import hudson.tools.ToolPropertyDescriptor;
import hudson.tools.ToolDescriptor;
import hudson.tools.ToolInstallation;
import hudson.tools.ToolInstaller;
import hudson.util.DescribableList;
import hudson.plugins.groovy.GroovyInstaller;
import hudson.plugins.groovy.GroovyInstallation;
/*
Installs Groovy on the node.
The idea was taken from: https://devops.lv/2016/12/05/jenkins-groovy-auto-installer/
and https://github.com/jenkinsci/jenkins-scripts/blob/master/scriptler/configMavenAutoInstaller.groovy
COMMENT 1: If we use this code directly (not as a separate method) then we get
java.io.NotSerializableException: hudson.plugins.groovy.GroovyInstaller
COMMENT 2: For some reason inst.getExecutable(channel) returns null. I use inst.forNode(node, null).getExecutable(channel) instead.
TODO: Check if https://jenkinsci.github.io/job-dsl-plugin/#method/javaposse.jobdsl.dsl.helpers.step.MultiJobStepContext.groovyCommand
works better.
*/
#NonCPS
def installGroovyOnSlave(String version) {
if ((version == null) || (version == "")) {
version = "2.4.7" // some default should be
}
/* Set up properties for our new Groovy installation */
def node = Jenkins.getInstance().slaves.find({it.name == env.NODE_NAME})
def proplist = new DescribableList<ToolProperty<?>, ToolPropertyDescriptor>()
def installers = new ArrayList<GroovyInstaller>()
def autoInstaller = new GroovyInstaller(version)
installers.add(autoInstaller)
def InstallSourceProperty isp = new InstallSourceProperty(installers)
proplist.add(isp)
def inst = new GroovyInstallation("Groovy", "", proplist)
/* Download and install */
autoInstaller.performInstallation(inst, node, null)
/* Define and add our Groovy installation to Jenkins */
def descriptor = Jenkins.getInstance().getDescriptor("hudson.plugins.groovy.Groovy")
descriptor.setInstallations(inst)
descriptor.save()
/* Output the current Groovy installation's path, to verify that it is ready for use */
def groovyInstPath = getGroovyExecutable(version)
println("Groovy " + version + " is installed in the node " + node.getDisplayName())
}
/* Returns the groovy executable path on the current node
If version is specified tries to find the specified version of groovy,
otherwise returns the first groovy installation that was found.
*/
#NonCPS
def getGroovyExecutable(String version=null) {
def node = Jenkins.getInstance().slaves.find({it.name == env.NODE_NAME})
def channel = node.getComputer().getChannel()
for (ToolInstallation tInstallation : Jenkins.getInstance().getDescriptor("hudson.plugins.groovy.Groovy").getInstallations()) {
if (tInstallation instanceof GroovyInstallation) {
if ((version == null) || (version == "")) {
// any version is appropriate for us
return tInstallation.forNode(node, null).getExecutable(channel)
}
// otherwise check for version
for (ToolProperty prop in tInstallation.getProperties()) {
if (prop instanceof InstallSourceProperty) {
for (ToolInstaller tInstaller: prop.installers) {
if (
(tInstaller instanceof GroovyInstaller) &&
(tInstaller.id.equals(version))
)
return tInstallation.forNode(node, null).getExecutable(channel)
}
}
}
}
}
return null
}
/* Wrapper function. Returns the groovy executable path as getGroovyExecutable()
but additionally tries to install if the groovy installation was not found.
*/
def getGroovy(String version=null) {
def installedGroovy = getGroovyExecutable(version)
if (installedGroovy != null) {
return installedGroovy
} else {
installGroovyOnSlave(version)
}
return getGroovyExecutable(version)
}
Just put these 3 methods to your pipeline script and you will be able to get the Groovy executable path with the help of the method getGroovy(). If it is not installed yet then the installation will be done automatically. You can test this code with the simple pipeline, like this:
// Main
parallel(
'Unix' : {
node ('build-unix') {
sh(getGroovy() + ' --version')
}
},
'Windows' : {
node ('build-win') {
bat(getGroovy() + ' --version')
}
}
)
For me the output was:
[build-unix] Groovy Version: 2.4.7 JVM: 1.8.0_222 Vendor: Private Build OS: Linux
[build-win] Groovy Version: 2.4.7 JVM: 11.0.1 Vendor: Oracle Corporation OS: Windows 10
To work with files on the slave workspace use the readFile, writeFile, findFiles etc steps.
Or if they are large as FloatingCoder said use native tooling; which may be running a groovy script.
A workaround could be load the library via sh command in Jenkinsfile.
So, if you use in Jenkinsfile:
sh 'groovy libraryName.groovy'
You can load the lib locally and in this way you can store File on the slave node.
Even without pipelines, there is no option to restrict a job based on slave agent label. So, I think, pipelines are only for master node execution.
Starting from release 2.4 of the Groovy plugin there is withGroovy step available which sets up the environment on the agent so that you can do sh 'groovy yourscript.groovy' with expected environments. It also enables limited interaction between Pipeline and groovy script.
See https://www.jenkins.io/doc/pipeline/steps/groovy/ for some details about the step.
I am running the following Groovy script which works perfect from the compiler & Jenkins ( if it runs a slave /node)
Because I want to run the script the "This project is parameterised" I noticed Jenkins is always running.
I Use the following Script:
// setup SSH connection:
sshString = "ssh -T -i keyfile -p 22 test#server.com "
cmdLine = "/appl/test/script.sh"
conString = sshString + cmdLine
// execute command
def proc = conString.execute()
def outputStream = new StringBuffer()
proc.waitForProcessOutput(outputStream, System.out)
output = (outputStream .toString())
println(output)
On the slave I get the result of the shell script, on the master the result is NULL.
What am I doing wrong
I am trying to get the current workspace of my Jenkins build using a Groovy pipeline script:
node('master') {
// PULL IN ENVIRONMENT VARIABLES
// Jenkins makes these variables available for each job it runs
def buildNumber = env.BUILD_NUMBER
def workspace = env.WORKSPACE
def buildUrl = env.BUILD_URL
// PRINT ENVIRONMENT TO JOB
echo "workspace directory is ${workspace}"
echo "build URL is ${env.BUILD_URL}"
}
It returns:
[Pipeline] Allocate node : Start
Running on master in /Users/Shared/Jenkins/Home/jobs/test/workspace
[Pipeline] node {
[Pipeline] echo
workspace directory is null
[Pipeline] echo
build URL is http://localhost:8080/job/test/5/
[Pipeline] } //node
[Pipeline] Allocate node : End
[Pipeline] End of Pipeline
Finished: SUCCESS
For me just ${WORKSPACE} worked without even initializing the variable workspace.
There is no variable included for that yet, so you have to use shell-out-read-file method:
sh 'pwd > workspace'
workspace = readFile('workspace').trim()
Or (if running on master node):
workspace = pwd()
I think you can also execute the pwd() function on the particular node:
node {
def PWD = pwd();
...
}
A quick note for anyone who is using bat in the job and needs to access Workspace:
It won't work.
$WORKSPACE https://issues.jenkins-ci.org/browse/JENKINS-33511 as mentioned here only works with PowerShell. So your code should have powershell for execution
stage('Verifying Workspace') {
powershell label: '', script: 'dir $WORKSPACE'
}
I have successfully used as shown below in Jenkinsfile:
steps {
script {
def reportPath = "${WORKSPACE}/target/report"
...
}
}
This is where you can find the answer in the job-dsl-plugin code.
Basically you can do something like this:
readFileFromWorkspace('src/main/groovy/com/groovy/jenkins/scripts/enable_safehtml.groovy')
In Jenkins pipeline script, I am using
targetDir = workspace
Works perfect for me. No need to use ${WORKSPACE}
The key is that, this works if used within double quotes instead of single quotes, below is my code and this worked!
script {
echo 'Entering Stage - Nexus Upload'
def artefactPath = "${WORKSPACE}/build/deploy/identityiq.war"
echo "printing the path ${artefactPath}"
}