How to get curl exit status from sh step command? - jenkins

I'm trying to get the result of a command into a boolean variable in my Jenkinsfile. The command is: curl -o/dev/null -sfI "$url", which can be used in sh like this:
if ( curl -o/dev/null -sfI "$url" ); then
echo "URL exists"
else
echo "URL does not exist"
fi
So, I need this condition in my jenkinsfile but I don't know how to recreate it. This is what I've tried:
def fileAlreadyExists = sh(
script: "curl -o/dev/null -sfI \"$url\"",
returnStdout: true
)
But seems to return false always.

Your command does not return any output due to -o /dev/null switch. If you want to catch exit code you would have to set returnStatus and not returnStdout option, like:
def fileAlreadyExists = sh(
script: "curl -o/dev/null -sfI \"$url\"",
returnStatus: true
)
Alternatively you could extend your Bash command to do echo $? after the curl command to echo last command exit code:
def fileAlreadyExists = sh(
script: "curl -o/dev/null -sfI \"$url\"; echo \\\$?",
returnStdout: true
) as Integer // explicit casting needed, because it returns String
The variable fileAlreadyExists stores an integer value, so you can use it in if () statement (Groovy evalutes if (0) to false, so if you expect 0 exit code then it is good idea to make this comparison explicit like:
if (fileAlreadyExists == 0) { /* exists */ } else { /* not exist */ }
As a side note: remember to escape \ if you want to pass it to the sh command - Jenkins strips single escape character, so if you want to pass e.g. double quote the underlying script then you have to escape it in the following way: \\\".

Related

how to escape double quotes in bash inside jenkins?

I have a Jenkins step as below.
stage('Initialize Namespace') {
when {
expression { params.initNS }
}
steps {
script {
setBuildStatus(processing_test_context, 'Initializing namespace', 'PENDING');
}
sh """ #!/bin/bash
set -e
namespaces="\${kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{\"\n\"}{end}'}"
echo "\${namespaces}"
if [[ \$namespaces =~ \$NAMESPACE ]]
then
kubectl create namespace $NAMESPACE || true
else
echo "The namespace already exists. Will be using the same"
fi
"""
}
How do I store the output of the command kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' in a variable? Currently, the error I am getting is ${kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{" "}{end}'}: bad substitution.
How do I escape " in the command?
Here is a working shell block for the script you provided.
sh"""
#!/bin/bash
namespaces=\$(kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{\"\\n\"}{end}')
echo "\$namespaces"
if [[ \$namespaces =~ \$NAMESPACE ]]
then
echo "Creating Namespace"
kubectl create namespace \$NAMESPACE || true
else
echo "The namespace already exists. Will be using the same"
fi
"""
If you don't need variable substitution(String interpolation) in your sh block you can use 3 single quotes rather than using double quotes which will allow you to get rid of multiple escape characters.
On a different note, the following is a different way to get what you need. Here $NAMESPACE has to be not empty always.
sh'''
#!/bin/bash
NS=$(kubectl get namespace $NAMESPACE --ignore-not-found);
echo $NS
if [[ -z "$NS" ]]; then
echo "Creating namespace $NAMESPACE"
kubectl create namespace $NAMESPACE || true
else
echo "The namespace already exists. Will be using the same"
fi
'''

In Jenkins, createItem triggers NotSerializableException, but does the job... should I be worried?

I call createItem in a function, from a loop derived from the list of git's currentBuild.changeSets
Here is the function:
def boolean shCreateJob(jobName) {
jobName = jobName.toString()
command = "curl -s -XPOST '$JENKINS_URL/createItem?name=$jobName' -u user:secret --data-binary ${TEMPLATE_FILE} -H 'Content-Type:text/xml'"
command = command.toString()
println("In shCreateJob, command=$command")
println("In shCreateJob, command.getClass()=${command.getClass()}")
try {
sh "curl $command"
}
catch (err) {
echo 'Exception during job creation : ' + err
}
return true
}
As you can see, in the function, I'm trying to cast the name and the command to String to ensure the sh command doesn't get anything exotic.
Here is the output:
In shCreateJob, command=curl -s -XPOST 'http://10.128.128.168:8080//createItem?name=cz_AAA' -u user:secret --data-binary #//var/lib/jenkins/jobs/_upload_template/config.xml -H 'Content-Type:text/xml'
[Pipeline] echo
In shCreateJob, command.getClass()=class java.lang.String
[Pipeline] sh
[Pipeline] echo
Exception during job creation : java.io.NotSerializableException: java.util.LinkedHashMap$Entry
And yet, the job ends in success, and also the new job is created as expected!
What's the deal?

Jenkinsfile issue while running script with sh

I have the following Jenkinsfile
def ARTIFACTS_JOB_ID;
def ARTIFACTS_URL;
node('CentOs') {
checkout scm
stage('Example') {
echo 'Download artifacts'
sh 'curljson --header "PRIVATE-TOKEN: tdzis3" "https://gitlab-sample.com/api/v4/projects/63/jobs" > jobs.json'
ARTIFACTS_JOB_ID = sh(returnStdout: true, script: 'python GetID.py').trim()
ARTIFACTS_URL = "https://gitlab-sample.com/api/v4/projects/63/jobs/${ARTIFACTS_JOB_ID}/artifacts"
echo "======> $ARTIFACTS_URL"
sh 'echo $ARTIFACTS_URL'
sh 'curl --output artifacts.zip --header "PRIVATE-TOKEN: tdzis3" "$ARTIFACTS_URL"'
}
}
while trying to get the artifact, it's calling
'curl --output artifacts.zip --header "PRIVATE-TOKEN: tdzis3" ""' with empty url
Similarly
echo "======> $ARTIFACTS_URL" //works fine
sh 'echo $ARTIFACTS_URL' // shows empty string, tried with ${ARTIFACTS_URL} aswell.
is there a way I can run it as below (so I don't have to use sh)
def ARTIFACTS_JOB_ID;
def ARTIFACTS_URL;
node('CentOs') {
checkout scm
stage('Example') {
script {
echo 'Download artifacts'
curljson --header "PRIVATE-TOKEN: tdzis3" "https://gitlab-sample.com/api/v4/projects/63/jobs" > jobs.json
}
}
}
without sh I am getting invalid syntax error while doing
curljson --header
The variable declaration ARTIFACTS_URL = "https://..." is a Groovy global variable. Hence, it is not naturally available to the sh step as a shell variable.
You need to wrap the commands inside the sh step with double quotes instead of single quotes as sh "echo $ARTIFACTS_URL" for Groovy to interpolate it as a Groovy variable.
Have you tried
sh "curl --output artifacts.zip --header 'PRIVATE-TOKEN: tdzis3' '{$ARTIFACTS_URL}'"
?
for multiple sh commands you could also use
sh """
curl command here
cd command next
etc
"""

Jenkins Groovy pass variables to parallel runs

I am having problems figuring out how to pass some variables into the parallel runs in the Jenkins groovy script below:
#!/usr/bin/env groovy
def call(version, project) {
sh '''#!/bin/bash
[[ ! -e ${WORKSPACE}/target/rpm/${project}/RPMS/ ]] && mkdir -p ${WORKSPACE}/target/rpm/${project}/RPMS/
(( $(ls ${WORKSPACE}/target/rpm/${project}/RPMS/*.rpm | wc -l) != 0 )) && rm ${WORKSPACE}/target/rpm/${project}/RPMS/*.rpm
cd ${WORKSPACE}/scripts/fpm_requirements && bundle install && bundle show fpm
'''
parallel (
"package foo": {
sh '''#!/bin/bash
export PATH=$PATH:~/bin:~/.gem/ruby/gems
cd ${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is $(echo ${version} | cut -d . -f 3)
'''
},
"package bar": {
sh '''#!/bin/bash
export PATH=$PATH:~/bin:~/.gem/ruby/gems
cd ${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is $(echo ${version} | cut -d . -f 3)
'''
}
)
}
So the version and project variables are populated in the first shell that is called but when they hit the two parallel runs they are not being pulled in.
I have tried a few different options to pass them in but none have worked.
Does anyone have any relevant ideas that might help?
You should change the ''' to """. In Groovy, string inside single/triple quote won't trigger string interpolation, but string inside single/triple double quote will do that.
So the ${version} and ${project} in your Shell script will be treated as variable from Shell context, but actually they are exist in Groovy context.
More about Groovy String at here, Below option 2 more suitable for your issue.
Option 1) using "" or """
"package foo": {
sh """#!/bin/bash
export PATH=\$PATH:~/bin:~/.gem/ruby/gems
cd \${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is \$(echo ${version} | cut -d . -f 3)
"""
},
"package bar": {
sh """#!/bin/bash
export PATH=\$PATH:~/bin:~/.gem/ruby/gems
cd \${WORKSPACE}/scripts/fpm_requirements
echo Project is ${project}
echo Version is ${version}
echo Iteration is \$(echo ${version} | cut -d . -f 3)
"""
}
Attention: need to escape the $ ahead of ${WORKSPACE} and $(echo ..), because we hope $ be kept after interpolation.
Option 2) using ' or ''' and inject version and project into Environment Variables of Shell context.
def call(version, project) {
env.version=version
env.project=project
// Groovy env api used to inject groovy value into environment variable
// so that you can refer groovy value later in shell script
// still use ''' in following code, no need to change
...

Jenkins pipeline sh does not seem to respect pipe in shell command

I am using a Jenkinsfile in a pipeline on version 2.32.2.
For various reasons I want to extract the version string from the pom. I was hoping I wouldn't have to add the maven help plugin and use evaluate.
I quickly came up with a little sed expression to get it out of the pom which uses pipes and works on the commandline in the jenkins workspace on the executor.
$ sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'
1.0.0-SNAPSHOT
It could probably be optimized, but I want to understand why the pipeline seems to be failing on piped sh commands. I've played with various string formats and am currently using a dollar slashy string.
The pipeline step looks like the following to allow for easy output of the command string:
script {
def ver_script = $/sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'/$
echo "${ver_script}"
POM_VERSION = sh(script: "${ver_script}", returnStdout: true)
echo "${POM_VERSION}"
}
When run in the jenkins pipeline I get the following console output where it seems to be separating the piped commands into separate commands:
[Pipeline] script
[Pipeline] {
[Pipeline] echo
sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'
[Pipeline] sh
[FRA-198-versioned-artifacts-44SD6DBQOGOI54UEF7NYE4ECARE7RMF7VQYXDPBVFOHS5CMSTFLA] Running shell script
+ sed -n /<version>/,/<version/p pom.xml
+ head -1
+ sed s/[[:blank:]]*<\/*version>//g
sed: couldn't write 89 items to stdout: Broken pipe
[Pipeline] }
[Pipeline] // script
Any guidance out there on how to properly use piped commands in a jenkinsfile ?
I finally put some thought into it and realized that pipe subshells are probably causing the issue. I know some of the evils of eval but I ended up wrappping this in an eval:
script {
def ver_script = $/eval "sed -n '/<version>/,/<version/p' pom.xml | head -1 | sed 's/[[:blank:]]*<\/*version>//g'"/$
echo "${ver_script}"
POM_VERSION = sh(script: "${ver_script}", returnStdout: true)
echo "${POM_VERSION}"
}
I know this kind of late answer, but whoever you who needs the solution without eval you can use /bin/bash -c "script" to make pipe works
script {
POM_VERSION = sh(script: "/bin/bash -c 'sed -n \'/<version>/,/<version/p\' pom.xml | head -1 | sed \'s/[[:blank:]]*<\/*version>//g\'\''", returnStdout: true)
echo "${POM_VERSION}"
}
The only problem with this method is hellish escape yet this way the subshell of pipe will be handled by our boy /bin/bash -c
If your environment allows it, I've found a simple solution to this problem to be to place your script containing pipes into a file, and then run that with sh, like so:
script.sh:
#!/bin/sh
kubectl exec --container bla -i $(kubectl get pods | awk '/foo-/{ print $1 }') -- php /code/dostuff
Jenkinsfile:
stage('Run script with pipes') {
steps {
sh "./script.sh"
}
}
The pipeline-utility-steps plugin nowadays includes a readMavenPom step, which allows to access the version as follows:
version = readMavenPom.getVersion()
So nothing detailed above worked for me using the scripted Jenkinsfile syntax with Groovy. I was able to get it working, however. The type of quotations you use are important. In the example below, I am trying to fetch the latest git tag from GitHub.
...
stage("Get latest git tag") {
if (env.CHANGE_BRANCH == 'master') {
sh 'git fetch --tags'
TAGGED_COMMIT = sh(script: 'git rev-list --branches=master --tags --max-count=1', returnStdout: true).trim()
LATEST_TAG = sh(script: 'git describe --abbrev=0 --tags ${TAGGED_COMMIT}', returnStdout: true).trim()
VERSION_NUMBER = sh(script: "echo ${LATEST_TAG} | cut -d 'v' -f 2", returnStdout: true).trim()
echo "VERSION_NUMBER: ${VERSION_NUMBER}"
sh 'echo "VERSION_NUMBER: ${VERSION_NUMBER}"'
}
}
...
Notice how the shell execution to assign LATEST_TAG works as expected (assigning the variable to v2.1.0). If we were to try the same thing (with single quotes) to assign VERSION_NUMBER, it would NOT work - the pipe messes everything up. Instead, we wrap the script in double quotes.
The first echo prints VERSION_NUMBER: 2.1.0 but the second prints VERSION_NUMBER:. If you want VERSION_NUMBER to be available in the shell commands, you have to assign the output of the shell command to env.VERSION_NUMBER as shown below:
...
stage("Get latest git tag") {
if (env.CHANGE_BRANCH == 'master') {
sh 'git fetch --tags'
TAGGED_COMMIT = sh(script: 'git rev-list --branches=master --tags --max-count=1', returnStdout: true).trim()
LATEST_TAG = sh(script: 'git describe --abbrev=0 --tags ${TAGGED_COMMIT}', returnStdout: true).trim()
env.VERSION_NUMBER = sh(script: "echo ${LATEST_TAG} | cut -d 'v' -f 2", returnStdout: true).trim()
echo "VERSION_NUMBER: ${VERSION_NUMBER}"
sh 'echo "VERSION_NUMBER: ${VERSION_NUMBER}"'
}
}
...
The first echo prints VERSION_NUMBER: 2.1.0 and the second prints VERSION_NUMBER: 2.1.0.
I am also struggling with the usage of pipe inside my jenkins pipeline but as a side note, if you want a simple way to extract the version of a maven pom, here's a very clean one I found in another post and that I'm using :
stage('Preparation') {
version = getVersion()
print "version : " + version
}
def getVersion() {
def matcher = readFile('pom.xml') =~ '<version>(.+)</version>'
matcher ? matcher[0][1] : null
}
gives you :
[Pipeline] echo
releaseVersion : 0.1.24
[Pipeline] sh

Resources