Jenkins - shell script - find parameter format not correct - jenkins

hope one of you can help - I'm running a script in jenkins pipeline, so that I can upload source maps to rollbar, so I need to loop through the minified js files - I'm trying to do this with the FIND command but it keeps giving me an error: find parameter format not correct - script below:
stages {
stage('Set correct environment link') {
steps {
script {
buildTarget = params.targetEnv
if(params.targetEnv.equals("prj1")){
linkTarget = "http://xxx.xxx.xx/xxx/"
} else if(params.targetEnv.equals(.......)..etc.
stage('Uploading source maps to Rollbar') {
steps {
sh ''' #!/bin/bash
# Save a short git hash, must be run from a git
# repository (or a child directory)
version=$(git rev-parse --short HEAD)
# Use the post_server_time access token, you can
# find one in your project access token settings
post_server_item=e1d04646bf614e039d0af3dec3fa03a7
echo "Uploading source maps for version $version!"
# We upload a source map for each resulting JavaScript
# file; the path depends on your build config
for path in $(find dist/renew -name "*.js"); do
# URL of the JavaScript file on the web server
url=${linkTarget}/$path
# a path to a corresponding source map file
source_map="#$path.map"
echo "Uploading source map for $url"
curl --silent --show-error https://api.rollbar.com/api/1/sourcemap \
-F access_token=$post_server_item \
-F version=$version \
-F minified_url=$url \
-F source_map=$source_map \
> /dev/null
done
'''
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
From Bash CLI

With the help of #jeb I managed to resolve this FIND issue on jenkins by placing the absolute path for (find) in the shell script - initially it was using the windows FIND cmd, so i needed to point to cygwin find cmd
before: for path in $(find dist/renew -name "*.js"); do
and after: for path in $(/usr/bin/find dist/renew -name "*.js"); do
Thaks to all that commented

Related

Script in Jenkins file giving exit code 1

I want to get a line from a file in my workspace. I am using this script :
stage('Test') {
steps {
script {
outputJenkins = 'output-jenkins.log'
sh "cd invoker && mvn clean install && mvn exec:java -Dexec.mainClass=\"com.JenkinsRunner\" -Dexec.args=\"qal ${GIT_COMMIT_HASH}\" > ../${outputJenkins}"
logFile = readFile(outputJenkins)
echo logFile
adminRepoLogLine = sh "echo logFile | grep \"Admin repo url is :::\""
echo adminRepoLogLine
}
}
}
But I am getting this error:
+ echo logFile
+ grep Admin repo url is :::
script returned exit code 1
The script works fine in my shell when I try it locally. Are there any contains around doing it in a JenkinsFile?
If we apply various fixes and improvements to the code in the question to achieve the desired functionality, then it will succeed:
stage('Test') {
steps {
script {
dir('invoker') {
sh(label: 'Maven Clean Install', script: 'mvn clean install')
// assign maven output to variable
String output = sh(label: 'Maven Git Log', script: "mvn exec:java -Dexec.mainClass=\"com.JenkinsRunner\" -Dexec.args=\"qal ${GIT_COMMIT_HASH}\"", returnStdout: true)
}
// assign regex return to variable
def adminRepoLogLine = output =~ /(.*Admin repo url is :::.*)/
// print extracted string from return
print adminRepoLogLine[0][1]
}
}
}
Note that GIT_COMMIT_HASH is neither an intrinsic Jenkins environment variable, nor defined in the pipeline code in the question, so it will need to be defined at Pipeline scope elsewhere in your code.
This is because the string literal logFile does not contain the string Admin repo url is :::. If there's no such match, then grep will exit with status 1.
You probably want to use
cat logFile | grep \"Admin repo url is :::\"
instead, or, even simpler:
grep \"Admin repo url is :::\" logFile
Append || true (or ||:) to the command if you want to avoid the errors when the log line does not appear.

Running liquibase update from script in jenkins pipeline

I'm trying to set up a Jenkins pipeline which will run a liquibase update whenever something is pushed to the master branch. The liquibase runner plugin for Jenkins has a security risk and therefore, I can't install it and run liquibase updates from that.
My liquibase* file (the bash script) is in my repository at the following path
/repo/liquibase/liquibase/liquibase*
I've set up the pipeline to run the following shell script. NOTE: I have the command set to liquibase --help for test purposes, but normally I'd want to run an update command.
export PATH=$PATH:/var/lib/jenkins/workspace/repo/liquibase
export PATH=$PATH:/var/lib/jenkins/workspace/repo/liquibase/liquibase
export PATH=$PATH:/var/lib/jenkins/workspace/repo/liquibase/liquibase/liquibase
export PATH=$PATH:/var/lib/jenkins/workspace/repo/liquibase/liquibase/jre/bin
cd liquibase
ls -ltr
chmod 755 liquibase/liquibase
chmod 755 liquibase/jre/bin/java.exe
liquibase --help
The liquibase --help command runs fine from the directory path /repo/liquibase in git bash. However, when I run it from Jenkins, I get the following error.
/var/lib/jenkins/workspace/Database_and_Repos/liquibase/liquibase/liquibase/jre/bin/java: No such file or directory
Build step 'Execute shell' marked build as failure
My liquibase file looks like this and it is the last line in the file that is causing the error.
#!/usr/bin/env bash
if [ ! -n "${LIQUIBASE_HOME+x}" ]; then
# echo "LIQUIBASE_HOME is not set."
## resolve links - $0 may be a symlink
PRG="$0"
while [ -h "$PRG" ] ; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG=`dirname "$PRG"`"/$link"
fi
done
LIQUIBASE_HOME=`dirname "$PRG"`
# make it fully qualified
LIQUIBASE_HOME=`cd "$LIQUIBASE_HOME" && pwd`
# echo "Liquibase Home: $LIQUIBASE_HOME"
fi
# build classpath from all jars in lib
if [ -f /usr/bin/cygpath ]; then
CP=.
for i in "$LIQUIBASE_HOME"/liquibase*.jar; do
i=`cygpath --windows "$i"`
CP="$CP;$i"
done
for i in "$LIQUIBASE_HOME"/lib/*.jar; do
i=`cygpath --windows "$i"`
CP="$CP;$i"
done
else
if [[ $(uname) = MINGW* ]]; then
CP_SEPARATOR=";"
else
CP_SEPARATOR=":"
fi
CP=.
for i in "$LIQUIBASE_HOME"/liquibase*.jar; do
CP="$CP""$CP_SEPARATOR""$i"
done
CP="$CP""$CP_SEPARATOR""$LIQUIBASE_HOME/lib/"
for i in "$LIQUIBASE_HOME"/lib/*.jar; do
CP="$CP""$CP_SEPARATOR""$i"
done
fi
if [ -z "${JAVA_HOME}" ]; then
#JAVA_HOME not set, try to find a bundled version
if [ -d "${LIQUIBASE_HOME}/jre" ]; then
JAVA_HOME="$LIQUIBASE_HOME/jre"
elif [ -d "${LIQUIBASE_HOME}/.install4j/jre.bundle/Contents/Home" ]; then
JAVA_HOME="${LIQUIBASE_HOME}/.install4j/jre.bundle/Contents/Home"
fi
fi
if [ -z "${JAVA_HOME}" ]; then
JAVA_PATH="$(which java)"
if [ -z "${JAVA_PATH}" ]; then
echo "Cannot find java in your path. Install java or use the JAVA_HOME environment variable"
fi
else
#Use path in JAVA_HOME
JAVA_PATH="${JAVA_HOME}/bin/java"
fi
# add any JVM options here
JAVA_OPTS="${JAVA_OPTS-}"
"${JAVA_PATH}" -cp "$CP" $JAVA_OPTS liquibase.integration.commandline.Main ${1+"$#"}
Has anyone run into this problem with liquibase commands in Jenkins? I've been googling all day, but haven't found much similar to this exact issue. Any help in the right direction would be great.
(We're updating the Liquibase Runner plugin. We have a release that is being reviewed for the security issues by the Jenkins team now.)
The error message seems to say that your "Execute shell" command in your changelog is not working correctly. Maybe the command is not installed, maybe it's calling a script that is not on your build machine.
One way to explore this to add an "echo" of the "Execute shell" command prior. Also, I'd pass --logLeve=DEBUG to Liquibase to get a better idea on the command it's trying to run.
Thanks for using Liquibase and Jenkins! I'll be talking about it here next month: https://www.cloudbees.com/devops-world/.
You could use the liquibase-maven-plugin and just call the maven phase in the pipeline:
sh mvn resources:resources liquibase:update
As for me, it is the best decision. Follow the official documentation

Groovy multiline shell script in Jenkins sh step does not return stdout

I am trying to save the output of a groovy shell script in a variable.
test = sh(returnStdout: true, script: "#!/bin/bash -l && export VAULT_ADDR=http://ourVault.de:8100 && export VAULT_SKIP_VERIFY=true && vault auth ${VAULT_TOKEN} && vault read -field=value test/${RELEASE2}/ID").trim()
But there is no output and I wonder why it does not capture the output?
If I do this:
def test = ""
sh"""#!/bin/bash -l
export VAULT_ADDR=http://ourVault.de:8100
export VAULT_SKIP_VERIFY=true
vault auth ${VAULT_TOKEN}
${test}=\"\$(vault read -field=value emea/test/hockey/ios/${RELEASE2}/appID)\"
"""
I see the output in the console. However, it doesn't get captured either. Is there any other way of capturing the output of multiline sh script?
The ${} syntax is not working that way. It can only be used add content to a string.
The returnStdout option can also be used with triple quoted scripts. So you probably want to do the following:
def test = sh returnStdout:true, script: """
#!/bin/bash -l
export VAULT_ADDR=http://ourVault.de:8100
export VAULT_SKIP_VERIFY=true
vault auth ${VAULT_TOKEN}
echo "\$(vault read -field=value emea/test/hockey/ios/${RELEASE2}/appID)" """

Jenkins 2.0: Running SBT in a docker container

I have the following Jenkinsfile:
def notifySlack = { String color, String message ->
slackSend(color: color, message: "${message}: Job ${env.JOB_NAME} [${env.BUILD_NUMBER}] (${env.BUILD_URL})")
}
node {
try {
notifySlack('#FFFF00', 'STARTED')
stage('Checkout project') {
checkout scm
}
scalaImage = docker.image('<myNexus>/centos-sbt:2.11.8')
stage('Test project') {
docker.withRegistry('<myNexus>', 'jenkins-nexus') {
scalaImage.inside('-v /var/lib/jenkins/.ivy2:/root/.ivy2') { c ->
sh 'sbt clean test'
}
}
}
if (env.BRANCH_NAME == 'master') {
stage('Release new version') {
docker.withRegistry('<myNexus>', 'jenkins-nexus') {
scalaImage.inside('-v /var/lib/jenkins/.ivy2:/root/.ivy2') { c ->
sh 'sbt release'
}
}
}
}
notifySlack('#00FF00', 'SUCCESSFUL')
} catch (e) {
currentBuild.result = "FAILED"
notifySlack('#FF0000', 'FAILED')
throw e
}
}
Unfortunately when I reach the sbt clean test line I end up with the following error:
java.lang.IllegalArgumentException: URI has a query component
at java.io.File.<init>(File.java:427)
at sbt.IO$.uriToFile(IO.scala:160)
at sbt.IO$.toFile(IO.scala:135)
at sbt.Classpaths$.sbt$Classpaths$$bootRepository(Defaults.scala:1942)
at sbt.Classpaths$$anonfun$appRepositories$1.apply(Defaults.scala:1912)
at sbt.Classpaths$$anonfun$appRepositories$1.apply(Defaults.scala:1912)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at sbt.Classpaths$.appRepositories(Defaults.scala:1912)
at sbt.Classpaths$$anonfun$58.apply(Defaults.scala:1193)
at sbt.Classpaths$$anonfun$58.apply(Defaults.scala:1190)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.EvaluateSettings$MixedNode.evaluate0(INode.scala:175)
at sbt.EvaluateSettings$INode.evaluate(INode.scala:135)
at sbt.EvaluateSettings$$anonfun$sbt$EvaluateSettings$$submitEvaluate$1.apply$mcV$sp(INode.scala:69)
at sbt.EvaluateSettings.sbt$EvaluateSettings$$run0(INode.scala:78)
at sbt.EvaluateSettings$$anon$3.run(INode.scala:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
If I run the simple docker run ... followed by docker exec I get what I want but I would like to work with the defined Jenkins functionality.
So this seems to be an SBT issue. I use version 0.13.16 inside the docker image. From what I understand the classpath contains a query parameter that SBT:
doesn't like
doesn't know how to handle
is illegal
I put no such query parameters myself so I thought that this .inside method does that. I checked the env in the container and found a single entry RUN_CHANGES_DISPLAY_URL=<my_ip>/job/scheduler/job/fix-jenkins-pipeline/23/display/redirect?page=changes. I tried to unset it but didn't manage to.
I'm out of ideas and am not really sure I'm in the right direction. Any help would be appreciated.
So after long and tedious searches what finally worked for me is setting explicitly the .sbt and .ivy2 folder like this inside the docker container:
sbt -Dsbt.global.base=.sbt -Dsbt.boot.directory=.sbt -Dsbt.ivy.home=.ivy2 clean test
That somehow prevents sbt from generating the ? folder and directly puts the aforementioned folders in the root of the directory checkout.
I spent a lot of time tracing this down through the code.
It looks like the easiest solution is to just pass -Duser.home=<path> to sbt, or to set it in the SBT_OPTS environment variable; then all the rest of the directories will be built as if the <path> is the user's home directory.
I fixed this by setting ivys cache directory -> How to override the location of Ivy's Cache?
The problem was that it wasn't set and by default it creates a ? folder which in return can't be handled by sbt itself.
I created a custom Dockerfile to have more control over sbt.
Here are the steps I executed to solve the issue:
I created a file called ivysettings.xml with the following contents:
<ivysettings>
<properties environment="env" />
<caches defaultCacheDir="/home/jenkins/.ivy2/cache" />
</ivysettings>
and a Dockerfile:
FROM openjdk:8
RUN wget -O- "http://downloads.lightbend.com/scala/2.11.11/scala-2.11.11.tgz" \
| tar xzf - -C /usr/local --strip-components=1
RUN curl -Ls https://git.io/sbt > /usr/bin/sbt && chmod 0755 /usr/bin/sbt
RUN adduser -u 1000 --disabled-password --gecos "" jenkins
ADD ./files/ivysettings.xml /home/jenkins/.ivy2/ivysettings.xml
RUN chown -R jenkins:jenkins /home/jenkins
USER jenkins
CMD ["sbt"]
I then pushed the image to our private docker repository and our pipeline finally works!
Problem is that jenkins is running a specific user inside the container. But overriding it does the trick.
withDockerContainer(args: "-u root -v ${HOME}/.sbt:/root/.sbt -v ${HOME}/.ivy2:/root/.ivy2 -e HOME=/root",
image: 'xyz/sbt:v') {

Echo off in Jenkins Console Output

I'm following guideline how to sign Android apk with Jenkins. I have parametrized Jenkins job with KSTOREPWD and KEYPWD. A part of Jenkins' job configuration (Build->Execute shell) is to take those parameters and store them as environment variables:
export KSTOREPWD=${KSTOREPWD}
export KEYPWD=${KEYPWD}
...
./gradlew assembleRelease
The problem is when the build is over anybody can access the build "Console Output" and see what passwords were entered; part of that output:
08:06:57 + export KSTOREPWD=secretStorePwd
08:06:57 + KSTOREPWD=secretStorePwd
08:06:57 + export KEYPWD=secretPwd
08:06:57 + KEYPWD=secretPwd
So I'd like to suppress echo before output from export commands and re-enable echo after export commands.
By default, Jenkins launches Execute Shell script with set -x. This causes all commands to be echoed
You can type set +x before any command to temporary override that behavior. Of course you will need set -x to start showing them again.
You can override this behaviour for the whole script by putting the following at the top of the build step:
#!/bin/bash +x
Here is an example of how to write the sh parameter in Jenkinsfile with no output in a more secure way, as suggested in official documentation. The set +x does the main magic as has been written in this answer.
The single-quotes will
cause the secret to be expanded by the shell as an environment
variable. The double-quotes are potentially less secure as the secret
is interpolated by Groovy, and so typical operating system process
listings (as well as Blue Ocean, and the pipeline steps tree in the
classic UI) will accidentally disclose it:
Insecure, wrong usage:
node {
withCredentials([string(credentialsId: 'mytoken', variable: 'TOKEN')]) {
sh /* WRONG! */ """
set +x
curl -H 'Token: $TOKEN' https://some.api/
"""
}
}
Correct usage ✅:
node {
withCredentials([string(credentialsId: 'mytoken', variable: 'TOKEN')]) {
sh '''
set +x
curl -H 'Token: $TOKEN' https://some.api/
'''
}
}
In your specific situation (using gradle and jenkins) you could also use a Password Parameter, using Gradle's pattern for environment variables (ORG_GRADLE_PROJECT_prop). Gradle will then set a propproperty on your project.
In your case this would look something like this
And you can use it in your gradle.properties like this
signingConfigs {
release {
storeFile file(KEYSTORE)
storePassword KSTOREPWD
keyAlias ALIAS
keyPassword KEYPWD
}
}
BTW - I recommend using the credentials binding plugin for KEYSTORE

Resources