I have a legacy project in Jenkins that hast to be pipelined (for
later parallelization), hence moving from simple tcsh script to
pipeline
running the script as
#!/bin/tcsh
source ./mysetting.sh
update
works but the same pipeline step fails due to missing alias expansion
stage ('update') {
steps {
//should be working but alias expansion fails
sh 'tcsh -c "source ./mysettings.sh; alias; update"'
//manually expanding the alias works fine
sh 'tcsh -c "source ./mysettings.sh; alias; python update.py;"'
}
}
calling alias in the steps properly lists all the set aliases, so I
can see them, but not use them.
I know in bash alias expansion has to be set
#enable shell option for alias_expansion
shopt -s expand_aliases
but in csh/tcsh that should be taken care of by source.
what am I missing?
found the solution:
sh '#!/bin/tcsh \n' +
'source ./mysettings.sh \n' +
'echo "Calling my alias" \n' +
'my_alias \n'
every line starting with sh launches a new shell, so it has to be in one line including line breaks.
further adding to the confusing was that documentation of jenkins says that it starts "a bash" but it launched /bin/sh which in my case pointed to something else
I am unable to properly pass a variable with spaces in Jenkinsfile to shell command
I have tried using quotes (double and single) backspaces and various other combination. Jenkins will treat the string like as 2 item and single quoted them.
The variable in question in the Jenkins file.
MYTIME = 2022-01-02 03:04:05
In Jenkinsfile
stage('list') {
steps{
sh script:'ansible-playbook -i ./hosts.ini ./ping_playbook.yml -e time=\\"$TIME\\" --limit ${ENV}', label: "ping test"
}
}
Jenkins will run it as
ansible-playbook -i ./lzhxjp-test-update/lzhxjp/hosts.ini .ping_playbook.yml -e 'time="2022-01-02' '11:22:33"' --limit sdktest
How do I put it so that Jenkins interpret it as -e time="2022-01-02 11:22:33"
Inside single-quoted strings no variables are expanded by Jenkins (Groovy to be exact).
Single-quoted strings are plain java.lang.String and don’t support interpolation.
https://groovy-lang.org/syntax.html#all-strings
Thus it is passed directly into the shell and therefore already subject to shell quoting, you can remove the \\ before the " and then the shell the will expand "$TIME" correctly.
I'm running a curl command in my Jenkinsfile.
post {
success {
script {
sh '''
|SCORE=+1
|GERRIT_COMMENT="$(cat <<-EOL
|Sonar result was: SUCCESS
|Report: ${Jenkins_URL}/job/${JOB_NAME}/${BUILD_NUMBER}/artifact/report1.txt
|EOL
|)"
|curl -s -u ${apiToken}: ${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children | json_pp -json_opt pretty,canonical > report1.txt
|echo "Voting unsuccessful"
'''.stripMargin().stripIndent()
archiveArtifacts artifacts: 'report1.txt', fingerprint: true
echo 'I Succeeded'
}
}
But I get the error
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/bin/json_pp
I can't use jq as it's not installed and installing it isn't an option.
The curl command works fine on my terminal but is failing in my Jenkins pipeline.
Also, when I do this instead, it works.
post {
success {
script {
sh '''
|SCORE=+1
|GERRIT_COMMENT="$(cat <<-EOL
|Sonar result was: SUCCESS
|Report: ${Jenkins_URL}/job/${JOB_NAME}/${BUILD_NUMBER}/artifact/report1.txt
|EOL
|)"
|echo "Voting unsuccessful"
'''.stripMargin().stripIndent()
sh """
curl -s -u ${apiToken}: '${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children' | json_pp -json_opt pretty,canonical > report1.txt
"""
archiveArtifacts artifacts: 'report1.txt', fingerprint: true
echo 'I Succeeded'
}
}
But it throws a warning in the console output.
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [apiToken]
What am I doing wrong, please?
In a Jenkins pipeline, how would you properly pass a JSON response using curl into a file?
I recommend to not use shell scripts whenever it is possible. Shell scripts are not cross platform and require installing additional tools (e.g. curl).
In your case the curl call could be replaced by the httpRequest step.
First let's replace the curl call and saves the result in a componentTree.json file:
httpRequest(
url: "${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children",
authorization: 'id-of-credentials-which-was-used-to-create-the-apiToken-variable',
outputFile: 'componentTree.json'
)
You want to format the JSON data in a human-readable format, so let's use the readJSON and writeJSON steps:
def json = readJSON(file: 'componentTree.json')
writeJSON(json: json, file: 'report1.txt', pretty: 4)
Now the report1.txt file contains JSON formatted with indent 4.
The componentTree.json file is written and read only once, so let's decrease the number of the IO operations:
def response = httpRequest(
url: "${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children",
authorization: 'id-of-credentials-which-was-used-to-create-the-apiToken-variable'
)
def json = readJSON(text: response.content)
writeJSON(json: json, file: 'report1.txt', pretty: 4)
About the warning:
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [apiToken]
Secrets never should be interpolated because they may contain special characters which could be interpreted. Example:
my secret: My' https://example.org; cat /etc/passwd; echo \
command: curl -u '${password}' https://private.server/path/file.txt
After the interpolation the following command is called:
curl -u 'My' https://example.org; cat /etc/passwd; echo \' https://private.server/path/file.txt
There are two options to fix it:
if apiToken is an environment variable:
sh "curl -s -u \$apiToken: '${Sonar_URL}/api/measures/component..."
if apiToken is a Groovy variable:
withEnv(["CREDENTIALS=${apiToken}"]) {
sh "curl -s -u \$CREDENTIALS: '${Sonar_URL}/api/measures/component..."
}
In both cases the dollar sign ($) is escaped before the credentials which means that shell script will resolve it (it will be taken from environment variables).
I am using Jenkins to take a number of parameters, generate an ansible-playbook command and run it. My Jenkins server is also my Ansible server.
My shell says ::
echo $ESXi_IP
echo $VM_NAME
echo $NIC1_MAC
echo $NIC2_MAC
echo $NIC3_MAC
echo $NIC4_MAC
echo $ESXi_HOSTNAME
echo $PLAYBOOK
ansible-playbook $PLAYBOOK --extra-vars "esxi_ip=$ESXi_IP vm_name=$VM_NAME nic1_mac=$NIC1_MAC nic2_mac=$NIC2_MAC nic3_mac=$NIC3_MAC nic4_mac=$NIC4_MAC esxi_hostname=$ESXi_HOSTNAME"
When I run the Job, the output is ::
+ ansible-playbook /root/ansible/sc-ece.yaml --extra-vars 'esxi_ip=5.232.66.49 vm_name=JenkinsTest nic1_mac=00:50:C0:A8:01:02 nic2_mac=00:50:0A:A9:37:A5 nic3_mac=00:50:0A:FF:FE:4C nic4_mac=00:50:AC:10:01:65 esxi_hostname=tmolab13-14iamesxi4'
ERROR! the playbook: /root/ansible/sc-ece.yaml could not be found
The playbook path is correct. there is no issue in it at all.
What seems to be missing here ?
You are correct Matt & Dave. Permissions to the folder was an issue. Thanks !
I'm following guideline how to sign Android apk with Jenkins. I have parametrized Jenkins job with KSTOREPWD and KEYPWD. A part of Jenkins' job configuration (Build->Execute shell) is to take those parameters and store them as environment variables:
export KSTOREPWD=${KSTOREPWD}
export KEYPWD=${KEYPWD}
...
./gradlew assembleRelease
The problem is when the build is over anybody can access the build "Console Output" and see what passwords were entered; part of that output:
08:06:57 + export KSTOREPWD=secretStorePwd
08:06:57 + KSTOREPWD=secretStorePwd
08:06:57 + export KEYPWD=secretPwd
08:06:57 + KEYPWD=secretPwd
So I'd like to suppress echo before output from export commands and re-enable echo after export commands.
By default, Jenkins launches Execute Shell script with set -x. This causes all commands to be echoed
You can type set +x before any command to temporary override that behavior. Of course you will need set -x to start showing them again.
You can override this behaviour for the whole script by putting the following at the top of the build step:
#!/bin/bash +x
Here is an example of how to write the sh parameter in Jenkinsfile with no output in a more secure way, as suggested in official documentation. The set +x does the main magic as has been written in this answer.
The single-quotes will
cause the secret to be expanded by the shell as an environment
variable. The double-quotes are potentially less secure as the secret
is interpolated by Groovy, and so typical operating system process
listings (as well as Blue Ocean, and the pipeline steps tree in the
classic UI) will accidentally disclose it:
Insecure, wrong usage:
node {
withCredentials([string(credentialsId: 'mytoken', variable: 'TOKEN')]) {
sh /* WRONG! */ """
set +x
curl -H 'Token: $TOKEN' https://some.api/
"""
}
}
Correct usage ✅:
node {
withCredentials([string(credentialsId: 'mytoken', variable: 'TOKEN')]) {
sh '''
set +x
curl -H 'Token: $TOKEN' https://some.api/
'''
}
}
In your specific situation (using gradle and jenkins) you could also use a Password Parameter, using Gradle's pattern for environment variables (ORG_GRADLE_PROJECT_prop). Gradle will then set a propproperty on your project.
In your case this would look something like this
And you can use it in your gradle.properties like this
signingConfigs {
release {
storeFile file(KEYSTORE)
storePassword KSTOREPWD
keyAlias ALIAS
keyPassword KEYPWD
}
}
BTW - I recommend using the credentials binding plugin for KEYSTORE