I am trying to run a script to dictate whether a Jenkins step runs. This script needs credentials, which we access by calling withCredentials.
Here's a slightly reduced example:
stage('Sample test stage') {
when {
expression {
withCredentials([usernamePassword(credentialsId: 'MY_CREDENTIALS_ID', usernameVariable: 'MY_USERNAME', passwordVariable: 'MY_TOKEN')]) {
return sh([returnStdout: true, script: 'node bin/my-script ${MY_TOKEN}']).trim().startsWith('success')
}
}
}
steps {
echo 'I should only run when the script passes'
}
}
However, this doesn't work. Jenkins returns the error: "syntax error: bad substitution."
I'm assuming it's down to the syntax used when trying to pass in the credentials. I've tried a couple of variations, such as returning withCredentials and wrapping the whole expression in withCredentials, but both of them had fairly dramatic errors.
I cannot find any documentation on this, which makes me wonder if it's illegal
I feel quite silly. There's a difference between " and ' in Groovy. Within single quotes, only Bash substitutions will work. Within double quotes, both Groovy and Bash substitutions will work.
Related
I'm attempting to run the following command in a shell block in my Jenkins pipeline:
jq '.Resources[].TargetService.Properties.TaskDefinition = "'"arn:aws:ecs:us-east-1:${ACCOUNT_NUMBER}:task-definition/${TASK_NAME}:${NEW_REVISION}"'"'
This command works perfectly fine when I run it directly on the Jenkins node in shell.
When I insert it into the Pipeline like this:
stage('process json') {
steps {
dir('mydir') {
sh """
NEW_REVISION=\$(cat revision.txt)
jq '.Resources[].TargetService.Properties.TaskDefinition = "'"arn:aws:ecs:us-east-1:\${env.AWS_ACCOUNT_NUMBER}:task-definition/\${env.TASK_NAME}:\${NEW_REVISION}"'"'
"""
}
}
}
I get a Bad substitution error without any more information. As far as I know, I'm escaping variables and quotation correctly. I can bypass the error if I remove the double quotes like this:
jq '.Resources[].TargetService.Properties.TaskDefinition = "arn:aws:ecs:us-east-1:${ACCOUNT_NUMBER}:task-definition/${TASK_NAME}:${NEW_REVISION}"'
But that ends up processing the variables literally.
Notes: I'm aware of the security issue by not passing jq --arg and prepared to modify my command after I can get the simpler format working. revision.txt contains a numeric value. The env.* variables are declared earlier as part of the pipeline environment.
env is a Jenkins Object and you seem to be escaping env.* variables as well. If you have already exported these variables as Environment variables they should be available to you in the shell environment. So simply drop the env part from the variables or remove the escape characters from such variables and let Jenkins interpolate them.
stage('process json') {
steps {
dir('mydir') {
sh """
NEW_REVISION=\$(cat revision.txt)
jq '.Resources[].TargetService.Properties.TaskDefinition = "'"arn:aws:ecs:us-east-1:\${AWS_ACCOUNT_NUMBER}:task-definition/\${TASK_NAME}:\${NEW_REVISION}"'"'
"""
}
}
}
I have checked many suggestions regarding the issue, but didn't find a suitable solution.
Initially my pipeline code was working fine but with a Jenkins security warning The following steps that have been detected may have insecure interpolation of sensitive variables. sh: [pass, user]
My code is as below:
withCredentials([usernamePassword(credentialsId: 'Jenkins-Nexus-tech', passwordVariable: 'pass', usernameVariable: 'user')]){
sh "wget --user=$user --password=$pass 'https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war' "
}
Warning:
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [pass, user]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
To avoid the Jenkins security warning as suggested in https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#interpolation-of-sensitive-environment-variables , I have modified my code as below. But with different permutation and combination of single and double quotes it is not working. It is throwing Bad substitution error:
withCredentials([usernamePassword(credentialsId: 'Jenkins-Nexus-tech', passwordVariable: 'pass', usernameVariable: 'user')]){
sh 'wget --user=$user --password=$pass \"https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war\" '
}
I have tried with 3 single quotes. with this I am not getting the warning but it is unable to substitute the variables in the wget URL:
withCredentials([usernamePassword(credentialsId: 'Jenkins-Nexus-tech', passwordVariable: 'pass', usernameVariable: 'user')]){
sh '''wget --user=$user --password=$pass 'https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war' '''
}
Error is as below as it can't substitute the values of the variable
[Pipeline] sh
+ wget --user=**** --password=**** https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war
HTTP request sent, awaiting response... 404 com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war
ERROR 404: com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war.
It would be really great if someone can assist here.
The suggestion outlined in the page you mention suggests you do not use Groovy string interpolation to pass sensitive data to sh steps. The suggestion is to let the sh script instead get the values from the environment.
However, in your case you still need the value held in the params.AuthorizationServerVersion in your sh step.
Groovy string interpolation is only used when the string is within double quotes. String interpolation is turned off when the string is in single quotes.
When a string is in double quotes, you can escape $s using a backslash. This will prevent inserting the contents of a variable into the string when you don't want this to occur.
What you can do, then, is use double quotes, escape the $s on the variables you want to read from the environment, and leave the $s un-escaped on the variables you want to access directly from your Groovy code.
An example, using the withCredentials plugin:
withCredentials([string(credentialsId: 'mytoken', variable: 'THE_TOKEN')]) {
sh "some_command --that-uses \$THE_TOKEN --as-an-environment-variable --and-also-uses $params.A_PARAMETER"
}
Here, the token is injected into the environment as a secret, and is used in the sh script by escaping the $. This means the actual Unix shell (bash, zsh, or whatever) will get the literal $ and expand it as specified by that shell language.
The $ for $params.A_PARAMETER is not escaped, and is within double quotes, so Groovy will perform string interpolation and replace $params.A_PARAMETER with the value of the parameter at runtime as the string is constructed in memory, before the string is passed as an argument to the sh step.
In your specific case, this is what you can use (within your withCredentials block, of course):
sh "wget --user=\$user --password=\$pass 'https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war'"
Note that the $ on $user and $pass are escaped, but the $ on ${params.AuthorizationServerVersion} is not escaped.
How do I access this variable in a shell script.
I have tried
echo params.$STATES;
echo $STATES;
Output for the first one is params. . Output for the second one is an empty string. The output I am expecting is the string I am passing when I build that job with parameters.
If you are using the 'Execute Shell' block in a Freestyle Project, you will need a $ (dollar sign) before the variable. Since you already tried this, there could be an issue not wrapping variable within {}
try echo ${STATES}
More info on curly braces around variables,
See: codeforester answer on usage of curly braces around shell variables
If you are using a Jenkinsfile without a shell block (valid in Scripted and Declarative)
Use dollar sign with double quotes
node(){
echo "$STATES"
}
pipeline {
agent any
stages {
stage('Hello') {
steps {
echo "$STATES"
}
}
}
}
or without Double quotes and dollar sign
node(){
echo STATES
}
pipeline {
agent any
stages {
stage('Hello') {
steps {
echo STATES
}
}
}
}
If you are using a Shell block in the Jenkinsfile pipeline, use
sh "echo $STATES" or sh "echo ${STATES}"
(as in the Freestyle Execute Shell block, Double quotes are needed for interpolation)
See
String interpolation in Jenkinsfiles
Handling parameters in Jenkinsfiles
In a Jenkinsfile I'm attempting to set an environment variable by setting the stdOut of a shell script. The script contains an AWS command that returns an InstanceID:
stage('Set InstanceID') {
steps {
script {
env.IID = sh (script: 'scripts/get-node-id.sh "${params.ENVIRONMENT}" "${params.NODE}"', returnStdout: true).trim()
}
}
}
No matter what I do or how many backslashes I use to escape the quotes, nothing works. I get a bad substitution error. I've also tried without double quotes.
If I hardcode in the shell script arguments, it runs fine.
How do I get this working if I want to use the parameter values here?
Groovy (the language of the Jenkinsfile) and Bash share the same substitution syntax. As you're using single quotes in your example code, the Groovy substitution does not work (see https://groovy-lang.org/syntax.html#_single_quoted_string). So Bash will try to do the substitution, but does not know these variables as they are Jenkins parameter values.
So solve this you need to use double quotes for your script and escape the double quotes in it (or use singe quotes):
stage('Set InstanceID') {
steps {
script {
env.IID = sh (script: "scripts/get-node-id.sh \"${params.ENVIRONMENT}\" \"${params.NODE}\"", returnStdout: true).trim()
}
}
}
I managed to resolve this, the comment from #yong above was almost what I needed. I needed three lots of double quotes and to unquote the variables:
stage('Set InstanceID') {
steps {
script {
env.IID = sh (script: """scripts/get-node-id.sh ${params.ENVIRONMENT} ${params.NODE}""", returnStdout: true).trim()
}
}
}
Given the following pipeline:
stages {
stage ("Checkout SCM") {
steps {
checkout scm
sh "echo ${CHANGE_AUTHOR_EMAIL}"
sh "echo ${CHANGE_ID}"
}
}
}
Why do these variables fail to resolve and provide a value?
Eventually I want to use these environment variables to send an email and merge a pull request:
post {
failure {
emailext (
attachLog: true,
subject: '[Jenkins] $PROJECT_NAME :: Build #$BUILD_NUMBER :: build failure',
to: '$CHANGE_AUTHOR_EMAIL',
replyTo: 'iadar#...',
body: '''<p>You are receiving this email because your pull request was involved in a failed build. Check the attached log file, or the console output at: $BUILD_URL to view the build results.</p>'''
)
}
}
and
sh "curl -X PUT -d '{\'commit_title\': \'Merge pull request\'}' <git url>/pulls/${CHANGE_ID}/merge?access_token=<token>"
Oddly enough, $PROJECT_NAME, $BUILD_NUMBER, $BUILD_URL do work...
Update: this may be an open bug... https://issues.jenkins-ci.org/browse/JENKINS-40486 :-(
Is there any workaround to get these values?
You need to be careful about how you refer to environment variables depending on whether it is shell or Groovy code, and how you are quoting.
When you do sh "echo ${CHANGE_ID}", what actually happens is that Groovy will interpolate the string first, by replacing ${CHANGE_ID} with the Groovy property CHANGE_ID, and that's where your error message is from. In Groovy, the environment variables are wrapped in env.
If you want to refer to the environment variables directly from your shell script, you either have to interpolate with env, use single quotes, or escape the dollar sign. All of the following should work:
sh 'echo $CHANGE_ID'
sh "echo \$CHANGE_ID"
sh "echo ${env.CHANGE_ID}"
For anyone who may come across this, these variables are available only if the checkbox for Build origin PRs (merged with base branch) was checked (this is in a multi-branch job).
See more in this other Jenkins issue: https://issues.jenkins-ci.org/browse/JENKINS-39838