How to use shell regular expression in jenkinsfile for jenkins pipeline? - jenkins

I am trying to replace the '/' from Git branch name with '_' in my jenkinsfile so that I can tag my docker image with the branch name. In bash the below command works fine
echo "${git_branch_name//\//_}"
But when use the above command in jenkinsfile as below it throws an error.
#!/usr/bin/env groovy
def commit_id
def imagetag
def branch_name
def git_branch_name
node('Nodename') {
stage('checkout') {
checkout (scm).$Branch_Param
sh "git rev-parse --short HEAD > .git/commit-id"
commit_id = readFile('.git/commit-id').trim()
sh "git rev-parse --abbrev-ref HEAD > .git/branch-name"
git_branch_name = readFile('.git/branch-name').trim()
branch_name= sh "echo ${git_branch_name//\//_}"
sh "echo ${commit_id}"
sh "echo ${branch_name}"
sh "echo Current branch is ${branch_name}"
}
}
WorkflowScript: 15: end of line reached within a simple string 'x' or "x" or /x/;
solution: for multi-line literals, use triple quotes '''x''' or """x""" or /x/ or $/x/$ # line 15, column 28.
sh "branch_name = echo ${git_branch_name//\//_}"
What am I doing wrong here? Should I use Groovy regular expression instead of shell? why is shell not being interpreted correctly?
Thank you

The issue is that you're asking Groovy itself to interpret the expression ${git_branch_name//\//_}, not the shell. Using double-quotes around the string you pass to the sh step is what causes that. So if you instead write the following, this first error will go away:
sh 'echo ${git_branch_name//\\//_}' // <- Note the single-quotes
Basically, always use single-quotes unless you specifically need to use groovy's string interpolation (see the very last echo at the bottom of this answer).
Interestingly, it seems when I tested I didn't need the shebang (#!/bin/bash) to specify bash as some comments suggest; this ${variable//x/y} replace syntax worked in an sh step as-is. I guess the shell spawned was bash. I don't know if that's always the case, or if our Jenkins box has been specifically setup that way.
Also note you need to escape the escape sequence ('\\/') because what you're passing to the sh step is a string literal in groovy code. If you don't add that extra backslash, the line passed to the shell to be interpreted by it will be echo ${git_branch_name////_}, which it won't understand.
But there are other issues as well. First, assigning the output of the sh step to branch_name as you do means branch_name will always equal null. To get the stdout from a line of shell code you need to pass the extra parameter returnStdout: true to sh:
branch_name = sh (
script: 'echo ${git_branch_name//\\//_}',
returnStdout: true
).trim () // You basically always need to use trim, because the
// stdout will have a newline at the end
For bonus points, we could wrap that sh call in a closure. I find myself using it often enough to make this a good idea.
// Get it? `sh` out, "shout!"
def shout = { cmd -> sh (script: cmd, returnStdout: true).trim () }
//...
branch_name = shout 'echo ${git_branch_name//\\//_}'
But finally, the major problem is that bash (or whatever shell is actually spawned) doesn't have access to groovy variables. As far as it knows, echo ${git_branch_name} outputs an empty string, and therefore so does echo ${git_branch_name//\//_}.
You have a couple of choices. You could skip the creation of .git/branch-name and just immediately output the string-replaced result of git rev-parse:
branch_name = shout 'name=$(git rev-parse --abbrev-ref HEAD) && echo ${name//\\//_}'
Or to simplify that further you could use groovy's string replace function rather than the bash syntax:
branch_name = shout ('git rev-parse --abbrev-ref HEAD').replace ('/', '_')
Personally, I find the latter quite a bit more readable. YMMV. So bringing it all together at last:
#!groovy
def shout = { cmd -> sh (script: cmd, returnStdout: true).trim () }
// Note that I'm not declaring any variables up here. They're not needed.
// But you can if you want, just to clearly declare the environment for
// future maintainers.
node ('Nodename') {
stage ('checkout') {
checkout (scm).$Branch_Param
commit_id = shout 'git rev-parse --short HEAD'
branch_name = shout ('git rev-parse --abbrev-ref HEAD').replace ('/', '_')
echo commit_id
echo branch_name
echo "The most recent commit on branch ${branch_name} was ${commit_id}"
}
}

Related

Script in Jenkins file giving exit code 1

I want to get a line from a file in my workspace. I am using this script :
stage('Test') {
steps {
script {
outputJenkins = 'output-jenkins.log'
sh "cd invoker && mvn clean install && mvn exec:java -Dexec.mainClass=\"com.JenkinsRunner\" -Dexec.args=\"qal ${GIT_COMMIT_HASH}\" > ../${outputJenkins}"
logFile = readFile(outputJenkins)
echo logFile
adminRepoLogLine = sh "echo logFile | grep \"Admin repo url is :::\""
echo adminRepoLogLine
}
}
}
But I am getting this error:
+ echo logFile
+ grep Admin repo url is :::
script returned exit code 1
The script works fine in my shell when I try it locally. Are there any contains around doing it in a JenkinsFile?
If we apply various fixes and improvements to the code in the question to achieve the desired functionality, then it will succeed:
stage('Test') {
steps {
script {
dir('invoker') {
sh(label: 'Maven Clean Install', script: 'mvn clean install')
// assign maven output to variable
String output = sh(label: 'Maven Git Log', script: "mvn exec:java -Dexec.mainClass=\"com.JenkinsRunner\" -Dexec.args=\"qal ${GIT_COMMIT_HASH}\"", returnStdout: true)
}
// assign regex return to variable
def adminRepoLogLine = output =~ /(.*Admin repo url is :::.*)/
// print extracted string from return
print adminRepoLogLine[0][1]
}
}
}
Note that GIT_COMMIT_HASH is neither an intrinsic Jenkins environment variable, nor defined in the pipeline code in the question, so it will need to be defined at Pipeline scope elsewhere in your code.
This is because the string literal logFile does not contain the string Admin repo url is :::. If there's no such match, then grep will exit with status 1.
You probably want to use
cat logFile | grep \"Admin repo url is :::\"
instead, or, even simpler:
grep \"Admin repo url is :::\" logFile
Append || true (or ||:) to the command if you want to avoid the errors when the log line does not appear.

Problem with escaping double quote Jenkinsfile

I start building a Jenkins job with a Jenkinsfile. In one of my steps, I create a variable. This variable echo's as expected, now I try to insert this variable into a new function and need to escape, everything looked fine until I see the last double quote on a new line and not on the end.
The line of code that create my variable is:
gitTag = sh(returnStdout: true, script: "git describe --tags--always")
The line code that use this variable:
sh "echo WEBSERVICE_VERSION=\"$gitTag\" > ${WORKSPACE}/webservice/src/webservice_version.py"
expected output WEBSERVICE_VERSION="$gitTag", but the output I see is WEBSERVICE_VERSION=$gitTag.
Did I make any mistake in escaping?
This should work for you
node('master'){
def gitTag="mytag"
sh "touch webservice_version.py"
sh "echo \\\"${gitTag}\\\" > webservice_version.py"
sh "cat webservice_version.py"
}

Passing variables extracted from shell in Jenkinsfile

I am trying to pass variables extracted in a stage in Jenkinsfile between stages. For example:
stage('Dummy Stage') {
sh '''#!/bin/bash -l
export abc=`output of some command`
.....
.....
'''
Now, how can I pass the variable abc to a subsequent stage? I have tried setting the variable by adding a def section at the top of the file but looks like it doesnt work. In the absence of a neater way, I am having to retype the commands
Here is what I do to get the number of commits on master as a global environment variable:
pipeline {
agent any
environment {
COMMITS_ON_MASTER = sh(script: "git rev-list HEAD --count", returnStdout: true).trim()
}
stages {
stage("Print commits") {
steps {
echo "There are ${env.COMMITS_ON_MASTER} commits on master"
}
}
}
}
You can use the longer form of the sh step and return the output (see Pipeline document). Your variable should be defined outside the stages.
You can use an imperatively created environment variable inside a script block in you stage steps, for example:
stage("Stage 1") {
steps {
script {
env.RESULT_ON_STAGE_1 = sh (
script: 'echo "Output of some command"',
returnStdout: true
)
}
echo "In stage 1: ${env.RESULT_ON_STAGE_1}"
}
}
stage("Stage 2") {
steps {
echo "In stage 2: ${env.RESULT_ON_STAGE_1}"
}
}
This guide explains use of environment variables in pipelines with examples.
My issue concerned having two 'sh' commands where one uses single quotes (where I set a variable) and the other uses double quotes (where I access 'env' variables set in the jenkinsfile such as BUILD_ID).
Here's how I solved it.
script {
env.TEST = sh(
script:
'''
echo "TEST"
''',
returnStdout: true
)
sh """
echo ${env.BUILD_ID}
echo ${env.TEST}
"""
}

CHANGE_AUTHOR_EMAIL and CHANGE_ID environment variables return "No such property: ..."

Given the following pipeline:
stages {
stage ("Checkout SCM") {
steps {
checkout scm
sh "echo ${CHANGE_AUTHOR_EMAIL}"
sh "echo ${CHANGE_ID}"
}
}
}
Why do these variables fail to resolve and provide a value?
Eventually I want to use these environment variables to send an email and merge a pull request:
post {
failure {
emailext (
attachLog: true,
subject: '[Jenkins] $PROJECT_NAME :: Build #$BUILD_NUMBER :: build failure',
to: '$CHANGE_AUTHOR_EMAIL',
replyTo: 'iadar#...',
body: '''<p>You are receiving this email because your pull request was involved in a failed build. Check the attached log file, or the console output at: $BUILD_URL to view the build results.</p>'''
)
}
}
and
sh "curl -X PUT -d '{\'commit_title\': \'Merge pull request\'}' <git url>/pulls/${CHANGE_ID}/merge?access_token=<token>"
Oddly enough, $PROJECT_NAME, $BUILD_NUMBER, $BUILD_URL do work...
Update: this may be an open bug... https://issues.jenkins-ci.org/browse/JENKINS-40486 :-(
Is there any workaround to get these values?
You need to be careful about how you refer to environment variables depending on whether it is shell or Groovy code, and how you are quoting.
When you do sh "echo ${CHANGE_ID}", what actually happens is that Groovy will interpolate the string first, by replacing ${CHANGE_ID} with the Groovy property CHANGE_ID, and that's where your error message is from. In Groovy, the environment variables are wrapped in env.
If you want to refer to the environment variables directly from your shell script, you either have to interpolate with env, use single quotes, or escape the dollar sign. All of the following should work:
sh 'echo $CHANGE_ID'
sh "echo \$CHANGE_ID"
sh "echo ${env.CHANGE_ID}"
For anyone who may come across this, these variables are available only if the checkbox for Build origin PRs (merged with base branch) was checked (this is in a multi-branch job).
See more in this other Jenkins issue: https://issues.jenkins-ci.org/browse/JENKINS-39838

Is it possible to capture the stdout from the sh DSL command in the pipeline

For example:
var output=sh "echo foo";
echo "output=$output";
I will get:
output=0
So, apparently I get the exit code rather than the stdout. Is it possible to capture the stdout into a pipeline variable, such that I could get:
output=foo
as my result?
Now, the sh step supports returning stdout by supplying the parameter returnStdout.
// These should all be performed at the point where you've
// checked out your sources on the slave. A 'git' executable
// must be available.
// Most typical, if you're not cloning into a sub directory
gitCommit = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()
// short SHA, possibly better for chat notifications, etc.
shortCommit = gitCommit.take(6)
See this example.
Note: The linked Jenkins issue has since been solved.
As mention in JENKINS-26133 it was not possible to get shell output as a variable. As a workaround suggested using of writ-read from temporary file. So, your example would have looked like:
sh "echo foo > result";
def output=readFile('result').trim()
echo "output=$output";
Try this:
def get_git_sha(git_dir='') {
dir(git_dir) {
return sh(returnStdout: true, script: 'git rev-parse HEAD').trim()
}
}
node(BUILD_NODE) {
...
repo_SHA = get_git_sha('src/FooBar.git')
echo repo_SHA
...
}
Tested on:
Jenkins ver. 2.19.1
Pipeline 2.4
You can try to use as well this functions to capture StdErr StdOut and return code.
def runShell(String command){
def responseCode = sh returnStatus: true, script: "${command} &> tmp.txt"
def output = readFile(file: "tmp.txt")
if (responseCode != 0){
println "[ERROR] ${output}"
throw new Exception("${output}")
}else{
return "${output}"
}
}
Notice:
&>name means 1>name 2>name -- redirect stdout and stderr to the file name
I had the same issue and tried almost everything then found after I came to know I was trying it in the wrong block. I was trying it in steps block whereas it needs to be in the environment block.
stage('Release') {
environment {
my_var = sh(script: "/bin/bash ${assign_version} || ls ", , returnStdout: true).trim()
}
steps {
println my_var
}
}
A short version would be:
echo sh(script: 'ls -al', returnStdout: true).result
def listing = sh script: 'ls -la /', returnStdout:true
Reference : http://shop.oreilly.com/product/0636920064602.do Page 433

Resources