Jenkins Groovy pass variables to parallel runs - jenkins

I am having problems figuring out how to pass some variables into the parallel runs in the Jenkins groovy script below:
#!/usr/bin/env groovy
def call(version, project) {
sh '''#!/bin/bash
[[ ! -e ${WORKSPACE}/target/rpm/${project}/RPMS/ ]] && mkdir -p ${WORKSPACE}/target/rpm/${project}/RPMS/
(( $(ls ${WORKSPACE}/target/rpm/${project}/RPMS/*.rpm | wc -l) != 0 )) && rm ${WORKSPACE}/target/rpm/${project}/RPMS/*.rpm
cd ${WORKSPACE}/scripts/fpm_requirements && bundle install && bundle show fpm
'''
parallel (
"package foo": {
sh '''#!/bin/bash
export PATH=$PATH:~/bin:~/.gem/ruby/gems
cd ${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is $(echo ${version} | cut -d . -f 3)
'''
},
"package bar": {
sh '''#!/bin/bash
export PATH=$PATH:~/bin:~/.gem/ruby/gems
cd ${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is $(echo ${version} | cut -d . -f 3)
'''
}
)
}
So the version and project variables are populated in the first shell that is called but when they hit the two parallel runs they are not being pulled in.
I have tried a few different options to pass them in but none have worked.
Does anyone have any relevant ideas that might help?

You should change the ''' to """. In Groovy, string inside single/triple quote won't trigger string interpolation, but string inside single/triple double quote will do that.
So the ${version} and ${project} in your Shell script will be treated as variable from Shell context, but actually they are exist in Groovy context.
More about Groovy String at here, Below option 2 more suitable for your issue.
Option 1) using "" or """
"package foo": {
sh """#!/bin/bash
export PATH=\$PATH:~/bin:~/.gem/ruby/gems
cd \${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is \$(echo ${version} | cut -d . -f 3)
"""
},
"package bar": {
sh """#!/bin/bash
export PATH=\$PATH:~/bin:~/.gem/ruby/gems
cd \${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is \$(echo ${version} | cut -d . -f 3)
"""
}
Attention: need to escape the $ ahead of ${WORKSPACE} and $(echo ..), because we hope $ be kept after interpolation.
Option 2) using ' or ''' and inject version and project into Environment Variables of Shell context.
def call(version, project) {
env.version=version
env.project=project
// Groovy env api used to inject groovy value into environment variable
// so that you can refer groovy value later in shell script
// still use ''' in following code, no need to change
...

Related

Export of a variable from a bash script

I've got to some variables which will be done with kubenetes command so I thought it best to put these in a bash script. I've managed to do that and called on it and see that variables get created but when it comes out of the bash script they are not assigned.
Within the Jenkinsfile script I have
steps {
sh '''
./bin/kube.sh
echo "Kube2 = ${SCRET}"
.....
and within the kube.sh file I have
#!/bin/bash
export SCRET=`kubectl -n keycloak get secret auser -o yaml | grep password | awk '{print $2}'`
echo "Kube2 = ${SCRET}"
I get the following results
+ ./bin/kube.sh
Kube1 = XXXXXXXX
+ echo 'SCRET = XXXXXXXX'
Kube2 =
Why is it that it gets unset again? What am I missing
Variables set in a subshell evaporate with that shell, and are not exported to the parent.
To set variables in the current environment using a script, you must source the code into the current context.
$: cat x
foo=bar
$: ./x && echo $foo # runs in a subshell - foo ends with ./x
$: . x && echo $foo # runs in current shell - foo is set
bar

Groovy shell script with a sed command in a Jenkins Pipeline

So writing Groovy with basic shell scripts seem to be much more difficult than it really should be.
I have a pipeline that needs to replace an entry in a file after running a packer command. It seems sensible to do this in the same shell script as the packer command as the variables are not available outside of the shell script even when exported.
The problem is that the sed command needs escape upon escape and still doesn't work. So this is what the Jenkins Pipeline Syntax generator suggested:
parallel (
"build my-application" : {
sh '''#!/bin/bash
export PATH=$PATH:~/bin
cd ${WORKSPACE}/platform/packer
packer build -machine-readable template.json | tee packer.out
AMI_APP=$(grep amazon-ebs,artifact,0,id,eu-west-2:ami- packer.out | awk -F: \'{ print $NF }\')
[[ ! ${AMI_APP} ]] && exit 1
sed -i.bak \'s!aws_ami_app = \\".*\\"!aws_ami_app = \\"\'"${AMI_APP}"\'\\"!\' ${WORKSPACE}/platform/terraform/env-${ENV}/env.auto.tfvars
'''
},
"build some-more-apps" : {
sh ''' *** same again different name ***
'''
}
)
What is the correct way to get a variable is a sed command working in a bash script running in groovy?
Any tips for the correct syntax going forward with Jenkins, groovy and bash - any documentation that actually helps?
EDIT
The original sed command that is running in a Jenkins Job shell is:
sed -i.bak 's!aws_ami_app = \".*\"!aws_ami_app = \"'"${AMI_APP}"'\"!' ${WORKSPACE}/platform/terraform/env-${ENV}/env.auto.tfvars
Because you put the shell script inside ''' which won't trigger Groovy String interpolation.
So you no need to escape any character, write the script as when you typing in Shell cmd window.
Below is example:
sh '''#!/bin/bash +x
echo "aws_ami_app = docker.xy.com/xy-ap123/conn:7et45u.1.23" > test.txt
echo "cpu = 512" >> test.txt
cat test.txt
AMI_APP=docker.xy.com/xy-ap123/conn:7et45u.1.25
sed -i 's,aws_ami_app.*,aws_ami_app = '"$AMI_APP"',' test.txt
cat test.txt
'''
Output in jenkins console:
[Pipeline] sh
[poc] Running shell script
aws_ami_app = docker.xy.com/xy-ap123/conn:7et45u.1.23
cpu = 512
aws_ami_app = docker.xy.com/xy-ap123/conn:7et45u.1.25
cpu = 512

extract version from package.json with bash inside jenkins pipeline

Script in package.json:
"scripts": {
"version": "echo $npm_package_version"
},
One of the stage in Jenkins pipeline:
stage('Build'){
sh 'npm install'
def packageVersion = sh 'npm run version'
echo $packageVersion
sh 'VERSION=${packageVersion} npm run build'
}
I got version from npm script output, but following line
echo $packageVersion
return null
Is packageVersion value not correctly assigned?
Edited:
when using Pipeline Utility Steps with following code
stage('Build'){
def packageJSON = readJSON file: 'package.json'
def packageJSONVersion = packageJSON.version
echo packageJSONVersion
sh 'VERSION=${packageJSONVersion}_${BUILD_NUMBER}_${BRANCH_NAME} npm run build'
}
I get
[Pipeline] echo
1.1.0
[Pipeline] sh
[...] Running shell script + VERSION=_16_SOME_BRANCH npm run build
So I am able to extract version, but still cannot pass it when running script
After your edit using readJSON, you now get string interpolation wrong. Variables within single quotes are not replaced in Groovy, only within double quotes.
sh 'VERSION=${packageJSONVersion}_${BUILD_NUMBER}_${BRANCH_NAME} npm run build'
must be
sh "VERSION=${packageJSONVersion}_${BUILD_NUMBER}_${BRANCH_NAME} npm run build"
The sh step by default returns nothing, so packageVersion should be null.
To return the output of the executed command, use it like this:
sh(script: 'npm run version', returnStdout: true)
This variant of sh returns the output instead of printing it.
Actually, I am wondering, why echo $packageVersion doesn't fail with an error, as this variable is not defined, but should be echo packageVersion.
For my it this:
sh(script: "grep \"version\" package.json | cut -d '\"' -f4 | tr -d '[[:space:]]'", returnStdout: true)
This worked for me:
Full version number:
PACKAGE_VERSION = sh returnStdout: true, script: '''grep 'version' package.json | cut -d '"' -f4 | tr '\n' '\0''''
echo "Current package version: $PACKAGE_VERSION"
$ > 1.2.3
Major version only:
PACKAGE_VERSION = sh returnStdout: true, script: '''grep 'version' package.json | cut -d '"' -f4 | cut -d '.' -f1 | tr '\n' '\0''''
echo "Current package Major version: $PACKAGE_VERSION"
$ > 1
stage('Read JSON') {
steps {
script {
def packageJson = readJSON file: 'package.json'
def packageVersion = packageJSON.version
echo "${packageJSONVersion}"
}
}
}
Below snippet worked for me: Credits to #Ferenc Takacs
version = sh(returnStdout: true, script: "grep 'version' package.json | cut -d '"' -f4 | tr '\n' '\0'")
This command will take exact property version: ... in package.json and can work on both Mac and Linux. The other solutions using grep will not give you correct answer in case of you have > 1 version keyword in your package.json (it'll return all of them instead of just the one you want)
awk -F'"' '/"version": ".+"/{ print $4; exit; }' package.json
To access an npm environment variable outside the scope of a run-script, parse the variable with bash:
$ npm run env | grep npm_package_version | cut -d '=' -f 2
Author: https://remarkablemark.org/blog/2018/08/14/package-json-version/

jenkins pipeline: multiline shell commands with pipe

I am trying to create a Jenkins pipeline where I need to execute multiple shell commands and use the result of one command in the next command or so. I found that wrapping the commands in a pair of three single quotes ''' can accomplish the same. However, I am facing issues while using pipe to feed output of one command to another command. For example
stage('Test') {
sh '''
echo "Executing Tests"
URL=`curl -s "http://localhost:4040/api/tunnels/command_line" | jq -r '.public_url'`
echo $URL
RESULT=`curl -sPOST "https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=$URL" | jq -r '.code'`
echo $RESULT
'''
}
Commands with pipe are not working properly. Here is the jenkins console output:
+ echo Executing Tests
Executing Tests
+ curl -s http://localhost:4040/api/tunnels/command_line
+ jq -r .public_url
+ URL=null
+ echo null
null
+ curl -sPOST https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=null
I tried entering all these commands in the jenkins snippet generator for pipeline and it gave the following output:
sh ''' echo "Executing Tests"
URL=`curl -s "http://localhost:4040/api/tunnels/command_line" | jq -r \'.public_url\'`
echo $URL
RESULT=`curl -sPOST "https://api.ghostinspector.com/v1/suites/[redacted]/execute/?apiKey=[redacted]&startUrl=$URL" | jq -r \'.code\'`
echo $RESULT
'''
Notice the escaped single quotes in the commands jq -r \'.public_url\' and jq -r \'.code\'. Using the code this way solved the problem
UPDATE: : After a while even that started to give problems. There were certain commands executing prior to these commands. One of them was grunt serve and the other was ./ngrok http 9000. I added some delay after each of these commands and it solved the problem for now.
The following scenario shows a real example that may need to use multiline shell commands. Which is, say you are using a plugin like Publish Over SSH and you need to execute a set of commands in the destination host in a single SSH session:
stage ('Prepare destination host') {
sh '''
ssh -t -t user#host 'bash -s << 'ENDSSH'
if [[ -d "/path/to/some/directory/" ]];
then
rm -f /path/to/some/directory/*.jar
else
sudo mkdir -p /path/to/some/directory/
sudo chmod -R 755 /path/to/some/directory/
sudo chown -R user:user /path/to/some/directory/
fi
ENDSSH'
'''
}
Special Notes:
The last ENDSSH' should not have any characters before it. So it
should be at the starting position of a new line.
use ssh -t -t if you have sudo within the remote shell command
I split the commands with &&
node {
FOO = world
stage('Preparation') { // for display purposes
sh "ls -a && pwd && echo ${FOO}"
}
}
The example outputs:
- ls -a (the files in your workspace
- pwd (location workspace)
- echo world

Jenkins pipeline sh does not seem to respect pipe in shell command

I am using a Jenkinsfile in a pipeline on version 2.32.2.
For various reasons I want to extract the version string from the pom. I was hoping I wouldn't have to add the maven help plugin and use evaluate.
I quickly came up with a little sed expression to get it out of the pom which uses pipes and works on the commandline in the jenkins workspace on the executor.
$ sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'
1.0.0-SNAPSHOT
It could probably be optimized, but I want to understand why the pipeline seems to be failing on piped sh commands. I've played with various string formats and am currently using a dollar slashy string.
The pipeline step looks like the following to allow for easy output of the command string:
script {
def ver_script = $/sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'/$
echo "${ver_script}"
POM_VERSION = sh(script: "${ver_script}", returnStdout: true)
echo "${POM_VERSION}"
}
When run in the jenkins pipeline I get the following console output where it seems to be separating the piped commands into separate commands:
[Pipeline] script
[Pipeline] {
[Pipeline] echo
sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'
[Pipeline] sh
[FRA-198-versioned-artifacts-44SD6DBQOGOI54UEF7NYE4ECARE7RMF7VQYXDPBVFOHS5CMSTFLA] Running shell script
+ sed -n /<version>/,/<version/p pom.xml
+ head -1
+ sed s/[[:blank:]]*<\/*version>//g
sed: couldn't write 89 items to stdout: Broken pipe
[Pipeline] }
[Pipeline] // script
Any guidance out there on how to properly use piped commands in a jenkinsfile ?
I finally put some thought into it and realized that pipe subshells are probably causing the issue. I know some of the evils of eval but I ended up wrappping this in an eval:
script {
def ver_script = $/eval "sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'"/$
echo "${ver_script}"
POM_VERSION = sh(script: "${ver_script}", returnStdout: true)
echo "${POM_VERSION}"
}
I know this kind of late answer, but whoever you who needs the solution without eval you can use /bin/bash -c "script" to make pipe works
script {
POM_VERSION = sh(script: "/bin/bash -c 'sed -n \'/<version>/,/<version/p\' pom.xml | head -1 | sed \'s/[[:blank:]]*<\/*version>//g\'\''", returnStdout: true)
echo "${POM_VERSION}"
}
The only problem with this method is hellish escape yet this way the subshell of pipe will be handled by our boy /bin/bash -c
If your environment allows it, I've found a simple solution to this problem to be to place your script containing pipes into a file, and then run that with sh, like so:
script.sh:
#!/bin/sh
kubectl exec --container bla -i $(kubectl get pods | awk '/foo-/{ print $1 }') -- php /code/dostuff
Jenkinsfile:
stage('Run script with pipes') {
steps {
sh "./script.sh"
}
}
The pipeline-utility-steps plugin nowadays includes a readMavenPom step, which allows to access the version as follows:
version = readMavenPom.getVersion()
So nothing detailed above worked for me using the scripted Jenkinsfile syntax with Groovy. I was able to get it working, however. The type of quotations you use are important. In the example below, I am trying to fetch the latest git tag from GitHub.
...
stage("Get latest git tag") {
if (env.CHANGE_BRANCH == 'master') {
sh 'git fetch --tags'
TAGGED_COMMIT = sh(script: 'git rev-list --branches=master --tags --max-count=1', returnStdout: true).trim()
LATEST_TAG = sh(script: 'git describe --abbrev=0 --tags ${TAGGED_COMMIT}', returnStdout: true).trim()
VERSION_NUMBER = sh(script: "echo ${LATEST_TAG} | cut -d 'v' -f 2", returnStdout: true).trim()
echo "VERSION_NUMBER: ${VERSION_NUMBER}"
sh 'echo "VERSION_NUMBER: ${VERSION_NUMBER}"'
}
}
...
Notice how the shell execution to assign LATEST_TAG works as expected (assigning the variable to v2.1.0). If we were to try the same thing (with single quotes) to assign VERSION_NUMBER, it would NOT work - the pipe messes everything up. Instead, we wrap the script in double quotes.
The first echo prints VERSION_NUMBER: 2.1.0 but the second prints VERSION_NUMBER:. If you want VERSION_NUMBER to be available in the shell commands, you have to assign the output of the shell command to env.VERSION_NUMBER as shown below:
...
stage("Get latest git tag") {
if (env.CHANGE_BRANCH == 'master') {
sh 'git fetch --tags'
TAGGED_COMMIT = sh(script: 'git rev-list --branches=master --tags --max-count=1', returnStdout: true).trim()
LATEST_TAG = sh(script: 'git describe --abbrev=0 --tags ${TAGGED_COMMIT}', returnStdout: true).trim()
env.VERSION_NUMBER = sh(script: "echo ${LATEST_TAG} | cut -d 'v' -f 2", returnStdout: true).trim()
echo "VERSION_NUMBER: ${VERSION_NUMBER}"
sh 'echo "VERSION_NUMBER: ${VERSION_NUMBER}"'
}
}
...
The first echo prints VERSION_NUMBER: 2.1.0 and the second prints VERSION_NUMBER: 2.1.0.
I am also struggling with the usage of pipe inside my jenkins pipeline but as a side note, if you want a simple way to extract the version of a maven pom, here's a very clean one I found in another post and that I'm using :
stage('Preparation') {
version = getVersion()
print "version : " + version
}
def getVersion() {
def matcher = readFile('pom.xml') =~ '<version>(.+)</version>'
matcher ? matcher[0][1] : null
}
gives you :
[Pipeline] echo
releaseVersion : 0.1.24
[Pipeline] sh

Resources