I'm using the Jenkins scripted pipeline and since the last update I get a new warning, which I want to silence, here is an MWE:
// FROM https://jenkins.io/doc/pipeline/examples/#parallel-multiple-nodes
def labels = []
if (Host == 'true') {
labels.add('<host-slavename>')
}
def builders = [:]
for (x in labels) {
def label = x
builders[label] = {
ansiColor('xterm') {
node(label) {
stage('cleanup') {
deleteDir()
}
stage('build') {
def timestamp = sh (script: 'echo -n `(date +%Y%m%d%H%M%S)`', returnStdout: true)
withCredentials([string(credentialsId: 'TEST_PASSWORD', variable: 'PASSWORD')]){
sh """
logger \
TEST_1 "${PASSWORD}" TEST_2 \
TEST_3 $timestamp TEST_4
"""
sh '''
logger \
TEST_1 "$PASSWORD" TEST_2 \
TEST_3 $timestamp TEST_4
'''
}
}
}
}
}
}
parallel builders
the first sh block returns
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [PASSWORD]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ logger TEST_1 **** TEST_2 TEST_3 20211029074911 TEST_4
which at least prints the timestamp and the password (it's censored, but works), but raises the warning.
the second sh block returns
+ logger TEST_1 **** TEST_2 TEST_3 TEST_4
So no warning, but also no timestamp.
Is there a way to write a scripted pipeline that shows no warning, but still the timestamp?
The warning occurs when you use Groovy string interpolation in the first sh step like "${PASSWORD}" for the reasons explained in Interpolation of Sensitive Environment Variables.
That's why you should always let the shell resolve environment variables as you correctly do in the 2nd sh step.
To use non-environment variables like timestamp, convert them to environment variables by wrapping the sh step in withEnv step:
withEnv(["timestamp=$timestamp"]) {
sh '''
logger \
TEST_1 "$PASSWORD" TEST_2 \
TEST_3 $timestamp TEST_4
'''
}
This limits the scope of the environment variable to the withEnv block.
Alternatively you could add a member to the env map:
env.timestamp = sh (script: 'echo -n `(date +%Y%m%d%H%M%S)`', returnStdout: true)
sh '''
logger \
TEST_1 "$PASSWORD" TEST_2 \
TEST_3 $timestamp TEST_4
'''
Related
I'm trying to create a pipeline that creates EC2 instances and I want to execute some shell commands on it.
After the Terraform applies the plan, AWS create new instances:
Build [no info about public IP yet]
Staging [no info about public IP yet]
It's impossible to retrieve public ipv4 information at the plan stage, therefore I created that construction:
stage ('Running command on a Build server') {
steps {
script {
BUILD_INSTANCE_IP = sh (
script: """
aws --region eu-central-1 ec2 describe-instances --filter \
"Name=instance-state-name,Values=running" --query \
"Reservations[*].Instances[*].[PublicIpAddress, Tags[?Key=='Name'].Value|[0]]" \
--output text | grep Build | cut -f1 //retrieve 'Build' instance IP
""", returnStdout: true
).trim()
sleep time: 3, unit: 'MINUTES' // wait 3 minutes until instance will be available to connect
sh ( script: """ssh -i ${AWS_KEY} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu#${BUILD_INSTANCE_IP} touch /home/ubuntu/test.txt""", returnStdout: true ).trim()
So, here I need to connect to the instance via sshagent {} directive, but after Jenkins moves to a new section, the data of variable 1 is reset to zero.
How do I make a variable that won't change?
I am sure this example will help you, How to use or re-use a variable across the stage throughout the pipeline
def x
node() {
stage('shell') {
x = sh(script: "echo 10", returnStdout: true) as int // needed to convert from String to int
print x // result = 10
}
stage('groovy') {
x = x + 10
print x // result = 20
}
}
In the below code I am fetching the version inside the script block, but I can not use it in the sh block and even not in the post block. Can someone please help. I am able to print the value in the println function. Variable is generating at the run time so I can not define inside the global environment block of Jenkinsfile. Can someone please help and tell me what am missing here?
stage('test build') {
steps {
script{
version = sh (
script: "cat ${WORKSPACE}/version.txt | grep var | awk -F ':' '{print \$2}'",
returnStdout: true
).trim()
}
println(version)
sh '''
echo "${version}"
'''
}
post {
success {
sh '''
echo "${version}"
'''
}
}
}
in my Mac, wget command working. How to fix this issue?
Error Message
wget
https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip
/Users/don/.jenkins/workspace/demo#tmp/durable-2702e009/script.sh:
line 1: wget: command not found
Full Pipeline Script
node('master') {
def home = sh(script: "echo $ANDROID_HOME",returnStdout: true).trim()
def SDKPath = "$home/Android/sdk"
stage("Preparing SDK"){
// Check SDK Downloaded
def isSDKDownloaded = sh(script: "test -e sdk-tools-linux-4333796.zip && echo true || echo false",returnStdout: true).trim()
if(isSDKDownloaded == "false"){
// Download SDK
sh "wget 'https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip'"
}
// Check if SDK is Extracted
def isExtracted = sh(script: "test -e $SDKPath/tools && echo true || echo false",returnStdout: true).trim()
if(isExtracted == "false"){
sh "mkdir -p $SDKPath"
//Unzip SDK
sh "unzip sdk-tools-linux-4333796.zip -d $SDKPath"
}
// Install SDK Tools
sh "yes | $SDKPath/tools/bin/sdkmanager 'build-tools;28.0.3' 'platform-tools' 'platforms;android-27'"
sh "ls $SDKPath/licenses"
// See installed And Available SDK
sh "$SDKPath/tools/bin/sdkmanager --list"
// Accept All SDK Licences
sh "yes | $SDKPath/tools/bin/sdkmanager --licenses"
}
def selectedBranch = SELECTED_RELEASE_BRANCH
stage('Checkout') {
git branch: selectedBranch, url: 'git#gitlab.com:o-apps/demo.git'
// Remove Existing local properties
sh 'rm local.properties ||:'
// Write sdk.dir Path into local properties file
sh "echo 'sdk.dir=$SDKPath' >> local.properties"
}
stage('Setup Tools') {
withCredentials([file(credentialsId: 'android_keystore', variable: 'KEYFILE')]) {
sh "cp \$KEYFILE app/key.jks"
}
}
stage('Build Release APK') {
sh "./gradlew clean assembleRelease"
}
stage('Upload to Play Store') {
androidApkUpload googleCredentialsId: 'key', apkFilesPattern: '**/*-release.apk', trackName: 'alpha'
}
stage('Cleanup Credential') {
sh "rm app/key.jks"
}
}
This is probably due to the $PATH environment variable which is different between your user and the user running Jenkins. Your user may be altering its $PATH by expanding it in the shell resource file (~/.bashrc, ~/.zshrc).
Not to worry, you can use the full path.
To find out the full path to wget, run this on the machine that runs the pipeline (the one labelled master):
% which wget
/usr/local/bin/wget
(Your path may naturally be different.)
Now use the full path:
// Download SDK
sh "/usr/local/bin/wget 'https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip'"
I am using a Jenkinsfile in a pipeline on version 2.32.2.
For various reasons I want to extract the version string from the pom. I was hoping I wouldn't have to add the maven help plugin and use evaluate.
I quickly came up with a little sed expression to get it out of the pom which uses pipes and works on the commandline in the jenkins workspace on the executor.
$ sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'
1.0.0-SNAPSHOT
It could probably be optimized, but I want to understand why the pipeline seems to be failing on piped sh commands. I've played with various string formats and am currently using a dollar slashy string.
The pipeline step looks like the following to allow for easy output of the command string:
script {
def ver_script = $/sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'/$
echo "${ver_script}"
POM_VERSION = sh(script: "${ver_script}", returnStdout: true)
echo "${POM_VERSION}"
}
When run in the jenkins pipeline I get the following console output where it seems to be separating the piped commands into separate commands:
[Pipeline] script
[Pipeline] {
[Pipeline] echo
sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'
[Pipeline] sh
[FRA-198-versioned-artifacts-44SD6DBQOGOI54UEF7NYE4ECARE7RMF7VQYXDPBVFOHS5CMSTFLA] Running shell script
+ sed -n /<version>/,/<version/p pom.xml
+ head -1
+ sed s/[[:blank:]]*<\/*version>//g
sed: couldn't write 89 items to stdout: Broken pipe
[Pipeline] }
[Pipeline] // script
Any guidance out there on how to properly use piped commands in a jenkinsfile ?
I finally put some thought into it and realized that pipe subshells are probably causing the issue. I know some of the evils of eval but I ended up wrappping this in an eval:
script {
def ver_script = $/eval "sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'"/$
echo "${ver_script}"
POM_VERSION = sh(script: "${ver_script}", returnStdout: true)
echo "${POM_VERSION}"
}
I know this kind of late answer, but whoever you who needs the solution without eval you can use /bin/bash -c "script" to make pipe works
script {
POM_VERSION = sh(script: "/bin/bash -c 'sed -n \'/<version>/,/<version/p\' pom.xml | head -1 | sed \'s/[[:blank:]]*<\/*version>//g\'\''", returnStdout: true)
echo "${POM_VERSION}"
}
The only problem with this method is hellish escape yet this way the subshell of pipe will be handled by our boy /bin/bash -c
If your environment allows it, I've found a simple solution to this problem to be to place your script containing pipes into a file, and then run that with sh, like so:
script.sh:
#!/bin/sh
kubectl exec --container bla -i $(kubectl get pods | awk '/foo-/{ print $1 }') -- php /code/dostuff
Jenkinsfile:
stage('Run script with pipes') {
steps {
sh "./script.sh"
}
}
The pipeline-utility-steps plugin nowadays includes a readMavenPom step, which allows to access the version as follows:
version = readMavenPom.getVersion()
So nothing detailed above worked for me using the scripted Jenkinsfile syntax with Groovy. I was able to get it working, however. The type of quotations you use are important. In the example below, I am trying to fetch the latest git tag from GitHub.
...
stage("Get latest git tag") {
if (env.CHANGE_BRANCH == 'master') {
sh 'git fetch --tags'
TAGGED_COMMIT = sh(script: 'git rev-list --branches=master --tags --max-count=1', returnStdout: true).trim()
LATEST_TAG = sh(script: 'git describe --abbrev=0 --tags ${TAGGED_COMMIT}', returnStdout: true).trim()
VERSION_NUMBER = sh(script: "echo ${LATEST_TAG} | cut -d 'v' -f 2", returnStdout: true).trim()
echo "VERSION_NUMBER: ${VERSION_NUMBER}"
sh 'echo "VERSION_NUMBER: ${VERSION_NUMBER}"'
}
}
...
Notice how the shell execution to assign LATEST_TAG works as expected (assigning the variable to v2.1.0). If we were to try the same thing (with single quotes) to assign VERSION_NUMBER, it would NOT work - the pipe messes everything up. Instead, we wrap the script in double quotes.
The first echo prints VERSION_NUMBER: 2.1.0 but the second prints VERSION_NUMBER:. If you want VERSION_NUMBER to be available in the shell commands, you have to assign the output of the shell command to env.VERSION_NUMBER as shown below:
...
stage("Get latest git tag") {
if (env.CHANGE_BRANCH == 'master') {
sh 'git fetch --tags'
TAGGED_COMMIT = sh(script: 'git rev-list --branches=master --tags --max-count=1', returnStdout: true).trim()
LATEST_TAG = sh(script: 'git describe --abbrev=0 --tags ${TAGGED_COMMIT}', returnStdout: true).trim()
env.VERSION_NUMBER = sh(script: "echo ${LATEST_TAG} | cut -d 'v' -f 2", returnStdout: true).trim()
echo "VERSION_NUMBER: ${VERSION_NUMBER}"
sh 'echo "VERSION_NUMBER: ${VERSION_NUMBER}"'
}
}
...
The first echo prints VERSION_NUMBER: 2.1.0 and the second prints VERSION_NUMBER: 2.1.0.
I am also struggling with the usage of pipe inside my jenkins pipeline but as a side note, if you want a simple way to extract the version of a maven pom, here's a very clean one I found in another post and that I'm using :
stage('Preparation') {
version = getVersion()
print "version : " + version
}
def getVersion() {
def matcher = readFile('pom.xml') =~ '<version>(.+)</version>'
matcher ? matcher[0][1] : null
}
gives you :
[Pipeline] echo
releaseVersion : 0.1.24
[Pipeline] sh
Using the Pipeline plugin in Jenkins 2.x, how can I access a Groovy variable that is defined somewhere at stage- or node-level from within a sh step?
Simple example:
node {
stage('Test Stage') {
some_var = 'Hello World' // this is Groovy
echo some_var // printing via Groovy works
sh 'echo $some_var' // printing in shell does not work
}
}
gives the following on the Jenkins output page:
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test Stage)
[Pipeline] echo
Hello World
[Pipeline] sh
[test] Running shell script
+ echo
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
As one can see, echo in the sh step prints an empty string.
A work-around would be to define the variable in the environment scope via
env.some_var = 'Hello World'
and print it via
sh 'echo ${env.some_var}'
However, this kind of abuses the environmental scope for this task.
To use a templatable string, where variables are substituted into a string, use double quotes.
sh "echo $some_var"
I am adding the comment from #Pedro as an answer because I think it is important.
For sh env vars we must use
sh "echo \$some_var"
You need to do something like below if a bash script is required :
Set this variable at global or local(function) level where from these can be accessible to sh script:
def stageOneWorkSpace = "/path/test1"
def stageTwoWorkSpace = "/path/test2"
In shell script call them like below
sh '''
echo ''' +stageOneWorkSpace+ '''
echo ''' +stageTwoWorkSpace+ '''
cp -r ''' +stageOneWorkSpace+'''/qa/folder1/* ''' +stageOneWorkSpace+'''/qa/folder2
'''
Make sure you start and end sh with three quotes like '''
I would like to add another scenario to this discussion.
I was using shell environment variables and groovy variables in the same script.
format='html'
for file in *.txt;
do mv -- "\$file" "\${file%.txt}.$format";
done
So here, What I have done is use \$ only for shell environment variables and use $ for groovy variables.
This is extension to #Dave Bacher's answer. I'm running multiple shell command in Groovy file & want to use output of one shell command to the next command as groovy variable. Using double quotes in shell command, groovy passes variable from one to another command but using single quotes it does not work, it returns null.
So use shell command like this in double quotes: sh "echo ${FOLDER_NAME}"
FOLDER_NAME = sh(script: $/
awk -F '=' '/CODE_COVERAGE_FOLDER/ {gsub("\"","");print$2}' ${WORKSPACE}/test.cfg
/$, returnStdout: true).trim()
echo "Folder: ${FOLDER_NAME}" // print folder name in groovy console
sh "mkdir -p ${WORKSPACE}/${FOLDER_NAME} && chmod 777 ${WORKSPACE}/${FOLDER_NAME}"