I'm trying to call a remote job from one Jenkins server to another, I have this working fine via a shell script. However, trying to translate it into a Jenkinsfile is causing me issues. The environment variable is always "null" when used inside of a stage, even thought this article says it should be globally available?
pipeline {
agent any
/* get crumb for CSRF */
environment {
def crumb = sh 'curl https://jenkins-remote/crumbIssuer/'
}
stages {
/* call remote job */
stage("remote") {
steps {
sh "curl -X POST -H ${env.crumb} https://jenkins-remote/buildWithParameters?foo"
}
}
}
}
The trimmed output looks like:
[remote_pipeline] Running shell script
+ curl -X POST -H null
I am using Jenkins v2.89.4, new "Pipeline" job, "pipeline script".
Thanks to #TrafLaf for pointing out the variable is null because it does not get set to the output of the curl command. My hacky solution was this:
environment {
def crumbRequest = sh 'curl https://jenkins-remote/crumbIssuer/ > crumbHeader'
crumbHeader = readFile('crumbHeader').trim()
}
As per the official documentation, This is how you define environment variables.
pipeline {
agent any
environment {
DISABLE_AUTH = 'true'
DB_ENGINE = 'sqlite'
}
stages {
stage('Build') {
steps {
echo "${env.DB_ENGINE}" # to access
}
}
}
}
But you have coded wrong,
environment {
def crumb = sh 'curl https://jenkins-remote/crumbIssuer/'
}
So please do the rest.
The sh task can now return output, so in theory the following should work (untested):
environment {
crumb = sh(script: 'curl https://jenkins-remote/crumbIssuer/',
returnStdout: true).trim()
}
Related
I am learning jenkins, and am working on a sample pipeline
pipeline {
agent any
stages {
stage('Stage1') {
steps {
bat '''
cd C:/Users/roger/project/
python -u script1.py
'''
}
}
stage('Stage2') {
steps {
bat '''
cd cd C:/Users/roger/project/abc/
python -u script2.py
'''
}
}
stage('Stage3') {
steps {
bat '''
cd cd C:/Users/roger/project/abc/new_dir/
python -u demo.py
'''
}
}
}
}
is there a way to store the base path of project C:/Users/roger/project/ as a variable, so that it can be used to append new path to it instead of writing the whole path.
How could I write above stages, so that I don't have to repeat writing the same base path each time to each stage
You have several options, the easiest way will be to define the parameter inside the environment directive (read more) which will make the parameter available for all stages in the pipeline and will also load it to the execution environment of any interpreter step like sh, bat and powershell thus making the parameter also available to the scripts you execute as an environment variable.
In addition the environment directive supports credential parameters which is very useful.
In your case it will look like:
pipeline {
agent any
environment {
BASE_PATH = 'C:/Users/roger/project/'
}
stages {
stage('Stage1') {
steps {
// Using the parameter as a runtime environment variable with bat syntax %%
bat '''
cd %BASE_PATH%
python -u script1.py
'''
}
}
stage('Stage2') {
steps {
// Using groovy string interpolation to construct the command with the parameter value
bat """
cd ${env.BASE_PATH}abc/
python -u script2.py
"""
}
}
}
}
Another option you have is to use global variables defined at the top section of the pipeline, which will behave like any groovy variable and will be available for all stages in your pipeline (but not for the execution environment of interpreter steps).
Something like:
BASE_PATH = 'C:/Users/roger/project/'
pipeline {
agent any
stages {
stage('Stage1') {
steps {
// Using the parameter insdie a dir step to change directory
dir(BASE_PATH) {
bat 'python -u script1.py'
}
}
}
stage('Stage2') {
steps {
// Using groovy string interpolation to construct the command with the parameter value
bat """
cd ${BASE_PATH}abc/
python -u script2.py
"""
}
}
}
}
This is related to one question I asked before: Using groovy to parse JSON object in shell scripts for Jenkin
basically I will need to pass a dynamic value as returned from sh scripts to an environment block, so that the following stage can re-use that value and pass the version as a label to JIRA plugin called Xray. But I aware that I cannot pass dynamic values to an environment block during the pipeline execution. So, I think I am going to need try a different route for that, not sure if anyone could give me some tips please?
def setLatestAppVersionLabel() {
def response = sh(script: "curl --silent ${APP_ARTIFACTORY_URL}/${XRAY_PLATFORM}/builds/latest.json", returnStdout: true).trim() as String
def jsonResponse = readJSON text: response
LATEST_VERSION = jsonResponse.id
echo "LATEST_VERSION -> ${LATEST_VERSION}"
}
JSON response looks like that:
{"id":"1.0.0-6",
"version":"1.0.0",
"build":6,
"tag":"android-v1.0.0-6",
"commitHash":"5a78c4665xxxxxxxxxxe1b62c682f84",
"dateCreated":"2020-03-02T08:11:29.912Z"}
and there is an environment block where I would like to pass the value to one of the variable defined there#
environment {
AWS_DEFAULT_REGION = 'uk-xxx'
XRAY_ENVIRONMENT = 'e2e'
VERSION_KEY = 'id'
XRAY_PLATFORM = 'Android'
APP_ARTIFACTORY_URL = 'https://artifactory.example.com/mobile'
LATEST_VERSION = ''
}
If this path not working, what else could I use? Want to re-use the latest version taken from JSON response for the next stage in the pipeline to use.
Next stage looks like this:
stage('Import Result to Xray') {
when {
expression { return fileExists('xxx-executor/target/AndroidxxxxE2EResults/cucumber-reports/Cucumber.json')}
}
steps {
xrayResultsImport('xxx-executor/target/AndroidxxxxxE2EResults/cucumber-reports/Cucumber.json', 'xxx_ANDROID_E2E_xxxxxxx_Tests', XRAY_LABELS, ['E2E', 'Android', LATEST_VERSION], env.BUILD_URL)
}
}
Sorry I have to put xxxx to make this question general due to project confidentiality.
To put it simple, you want to use the version you fetched from a JSON response and want to use it in all stages of your Jenkins pipeline.
Ensure you've jq utility installed in your jenkins agent.
pipeline {
agent any
environment {
XRAY_LATEST_VERSION = ''
}
stages {
stage(‘Get Version ') {
steps {
script {
XRAY_LATEST_VERSION = sh(script: 'curl -s ${APP_ARTIFACTORY_URL}/${XRAY_PLATFORM}/builds/latest.json | jq .version | sed \'s/"//g\'', returnStdout: true).trim()
}
}
}
stage('Print'){
steps{
echo "${XRAY_LATEST_VERSION}"
}
}
}
}
You can use the variable ${XRAY_LATEST_VERSION} in any stages you want the and the value will be rendered across.
I have a pipeline job that runs a maven build. In the "post" section of the pipeline, I want to get the log file so that I can perform some failure analysis on it using some regexes. I have tried the following:
def logContent = Jenkins.getInstance()
.getItemByFullName(JOB_NAME)
.getBuildByNumber(
Integer.parseInt(BUILD_NUMBER))
.logFile.text
Error for the above code
Scripts not permitted to use staticMethod jenkins.model.Jenkins
getInstance
currentBuild.rawBuild.getLogFile()
Error for the above code
Scripts not permitted to use method hudson.model.Run getLogFile
From my research, when I encounter these, I should be able to go to the scriptApproval page and see a prompt to approve these scripts, but when I go to that page, there are no new prompts.
I've also tried loading the script in from a separate file and running it on a different node with no luck.
I'm not sure what else to try at this point, so that's why I'm here. Any help is greatly appreciated.
P.S. I'm aware of the BFA tool, and I've tried manually triggering the analysis early, but in order to do that, I need to be able to access the log file, so I run into the same issue.
You can use pipeline step httpRequest from here
pipeline {
agent any
stages {
stage('Build') {
steps {
echo 'Test fetch build log'
}
post {
always {
script {
def logUrl = env.BUILD_URL + 'consoleText'
def response = httpRequest(
url: logUrl,
authentication: '<credentialsId of jenkins user>',
ignoreSslErrors: true
)
def log = response.content
echo 'Build log: ' + log
}
}
}
}
}
}
If your jenkins job can run on linux machine, you can use curl to archive same goal.
pipeline {
agent any
stages {
stage('Build') {
environment {
JENKINS_AUTH = credentials('< credentialsId of jenkins user')
}
steps {
sh 'pwd'
}
post {
always {
script {
def logUrl = env.BUILD_URL + 'consoleText'
def cmd = 'curl -u ${JENKINS_AUTH} -k ' + logUrl
def log = sh(returnStdout: true, script: cmd).trim()
echo 'Build log: ' +
echo log
}
}
}
}
}
}
Above two approaches both require the credentials is Username and password format. More detail about what is it and how to add in Jenkins, please look at here
Currently this is not possible via the RunWrapper object that is made available. See https://issues.jenkins.io/browse/JENKINS-46376 for a request to add this.
So the only options are:
explicitly whitelisting the methods
read the log via the URL as described in the other answer, but this requires either anonymous read access or using proper credentials.
I want to pass a variable which I read in stage A towards stage B somehow. I see in some examples that people write it to a file, but I guess that is not really a nice solution. I tried writing it to an environment variable, but I'm not really successful on that. How can I set it up properly?
To get it working I tried a lot of things and read that I should use the """ instead of ''' to start a shell and escape those variables to \${foo} for example.
Below is what I have as a pipeline:
#!/usr/bin/env groovy
pipeline {
agent { node { label 'php71' } }
environment {
packageName='my-package'
packageVersion=''
groupId='vznl'
nexus_endpoint='http://nexus.devtools.io'
nexus_username='jenkins'
nexus_password='J3nkins'
}
stages{
// Package dependencies
stage('Install dependencies') {
steps {
sh '''
echo Skip composer installation
#composer install --prefer-dist --optimize-autoloader --no-interaction
'''
}
}
// Unit tests
stage('Unit Tests') {
steps {
sh '''
echo Running PHP code coverage tests...
#composer test
'''
}
}
// Create artifact
stage('Package') {
steps {
echo 'Create package refs'
sh """
mkdir -p ./build/zpk
VERSIONTAG=\$(grep 'version' composer.json)
REGEX='"version": "([0-9]+.[0-9]+.[0-9]+)"'
if [[ \${VERSIONTAG} =~ \${REGEX} ]]
then
env.packageVersion=\${BASH_REMATCH[1]}
/usr/bin/zs-client packZpk --folder=. --destination=./build/zpk --name=${env.packageName}-${env.packageVersion}.zpk --version=${env.packageVersion}
else
echo "No version found!"
exit 1
fi
"""
}
}
// Publish ZPK package to Nexus
stage('Publish packages') {
steps {
echo "Publish ZPK Package"
sh "curl -u ${env.nexus_username}:${env.nexus_password} --upload-file ./build/zpk/${env.packageName}-${env.packageVersion}.zpk ${env.nexus_endpoint}/repository/zpk-packages/${groupId}/${env.packageName}-${env.packageVersion}.zpk"
archive includes: './build/**/*.{zpk,rpm,deb}'
}
}
}
}
As you can see the packageVersion which I read from stage Package needs to be used in stage Publish as well.
Overall tips against the pipeline are of course always welcome as well.
A problem in your code is that you are assigning version of environment variable within the sh step. This step will execute in its own isolated process, inheriting parent process environment variables.
However, the only way of passing data back to the parent is through STDOUT/STDERR or exit code. As you want a string value, it is best to echo version from the sh step and assign it to a variable within the script context.
If you reuse the node, the script context will persist, and variables will be available in the subsequent stage. A working example is below. Note that any try to put this within a parallel block can be of failure, as the version information variable can be written to by multiple processes.
#!/usr/bin/env groovy
pipeline {
environment {
AGENT_INFO = ''
}
agent {
docker {
image 'alpine'
reuseNode true
}
}
stages {
stage('Collect agent info'){
steps {
echo "Current agent info: ${env.AGENT_INFO}"
script {
def agentInfo = sh script:'uname -a', returnStdout: true
println "Agent info within script: ${agentInfo}"
AGENT_INFO = agentInfo.replace("/n", "")
env.AGENT_INFO = AGENT_INFO
}
}
}
stage("Print agent info"){
steps {
script {
echo "Collected agent info: ${AGENT_INFO}"
echo "Environment agent info: ${env.AGENT_INFO}"
}
}
}
}
}
Another option which doesn't involve using script, but is just declarative, is to stash things in a little temporary environment file.
You can then use this stash (like a temporary cache that only lives for the run) if the workload is sprayed out across parallel or distributed nodes as needed.
Something like:
pipeline {
agent any
stages {
stage('first stage') {
steps {
// Write out any environment variables you like to a temporary file
sh 'echo export FOO=baz > myenv'
// Stash away for later use
stash 'myenv'
}
}
stage ("later stage") {
steps {
// Unstash the temporary file and apply it
unstash 'myenv'
// use the unstashed vars
sh 'source myenv && echo $FOO'
}
}
}
}
For example:
var output=sh "echo foo";
echo "output=$output";
I will get:
output=0
So, apparently I get the exit code rather than the stdout. Is it possible to capture the stdout into a pipeline variable, such that I could get:
output=foo
as my result?
Now, the sh step supports returning stdout by supplying the parameter returnStdout.
// These should all be performed at the point where you've
// checked out your sources on the slave. A 'git' executable
// must be available.
// Most typical, if you're not cloning into a sub directory
gitCommit = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()
// short SHA, possibly better for chat notifications, etc.
shortCommit = gitCommit.take(6)
See this example.
Note: The linked Jenkins issue has since been solved.
As mention in JENKINS-26133 it was not possible to get shell output as a variable. As a workaround suggested using of writ-read from temporary file. So, your example would have looked like:
sh "echo foo > result";
def output=readFile('result').trim()
echo "output=$output";
Try this:
def get_git_sha(git_dir='') {
dir(git_dir) {
return sh(returnStdout: true, script: 'git rev-parse HEAD').trim()
}
}
node(BUILD_NODE) {
...
repo_SHA = get_git_sha('src/FooBar.git')
echo repo_SHA
...
}
Tested on:
Jenkins ver. 2.19.1
Pipeline 2.4
You can try to use as well this functions to capture StdErr StdOut and return code.
def runShell(String command){
def responseCode = sh returnStatus: true, script: "${command} &> tmp.txt"
def output = readFile(file: "tmp.txt")
if (responseCode != 0){
println "[ERROR] ${output}"
throw new Exception("${output}")
}else{
return "${output}"
}
}
Notice:
&>name means 1>name 2>name -- redirect stdout and stderr to the file name
I had the same issue and tried almost everything then found after I came to know I was trying it in the wrong block. I was trying it in steps block whereas it needs to be in the environment block.
stage('Release') {
environment {
my_var = sh(script: "/bin/bash ${assign_version} || ls ", , returnStdout: true).trim()
}
steps {
println my_var
}
}
A short version would be:
echo sh(script: 'ls -al', returnStdout: true).result
def listing = sh script: 'ls -la /', returnStdout:true
Reference : http://shop.oreilly.com/product/0636920064602.do Page 433