I am having these lines in my Jenkinsfile:
parameters {
string(name: 'DATABASE', defaultValue: 'jenkinsdatabase',
description: 'The name of the database')
}
(...)
Now I want to use the value of ${params.DATABASE} in a step of a stage e.g.
sh 'mysql --user ${USER} -p${PASSWORD} --host ${HOST} -e "DROP DATABASE IF EXISTS ${params.DATABASE};CREATE DATABASE ${params.DATABASE} DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; commit;";export GRADLE_OPTS="-Xms1536m -Xmx1536m"'
But this ends with a exeception: Bad substitution
Can anybody help me?
Your sh script body uses single quotes around it.
sh '...'
Since these are single quotes, string interpolation will not occur. This means that all of your $ strings will use the shell variable substitution instead of the Groovy substitution. You are getting a Bad substitution in the shell because the shell won't be able to substitute ${params.DATABASE} because that is an invalid shell substitution.
params is a Groovy variable, so switching your single quotes to double quotes will perform string interpolation of the variables.
sh "..."
Related
I'm attempting to run the following command in a shell block in my Jenkins pipeline:
jq '.Resources[].TargetService.Properties.TaskDefinition = "'"arn:aws:ecs:us-east-1:${ACCOUNT_NUMBER}:task-definition/${TASK_NAME}:${NEW_REVISION}"'"'
This command works perfectly fine when I run it directly on the Jenkins node in shell.
When I insert it into the Pipeline like this:
stage('process json') {
steps {
dir('mydir') {
sh """
NEW_REVISION=\$(cat revision.txt)
jq '.Resources[].TargetService.Properties.TaskDefinition = "'"arn:aws:ecs:us-east-1:\${env.AWS_ACCOUNT_NUMBER}:task-definition/\${env.TASK_NAME}:\${NEW_REVISION}"'"'
"""
}
}
}
I get a Bad substitution error without any more information. As far as I know, I'm escaping variables and quotation correctly. I can bypass the error if I remove the double quotes like this:
jq '.Resources[].TargetService.Properties.TaskDefinition = "arn:aws:ecs:us-east-1:${ACCOUNT_NUMBER}:task-definition/${TASK_NAME}:${NEW_REVISION}"'
But that ends up processing the variables literally.
Notes: I'm aware of the security issue by not passing jq --arg and prepared to modify my command after I can get the simpler format working. revision.txt contains a numeric value. The env.* variables are declared earlier as part of the pipeline environment.
env is a Jenkins Object and you seem to be escaping env.* variables as well. If you have already exported these variables as Environment variables they should be available to you in the shell environment. So simply drop the env part from the variables or remove the escape characters from such variables and let Jenkins interpolate them.
stage('process json') {
steps {
dir('mydir') {
sh """
NEW_REVISION=\$(cat revision.txt)
jq '.Resources[].TargetService.Properties.TaskDefinition = "'"arn:aws:ecs:us-east-1:\${AWS_ACCOUNT_NUMBER}:task-definition/\${TASK_NAME}:\${NEW_REVISION}"'"'
"""
}
}
}
I am unable to properly pass a variable with spaces in Jenkinsfile to shell command
I have tried using quotes (double and single) backspaces and various other combination. Jenkins will treat the string like as 2 item and single quoted them.
The variable in question in the Jenkins file.
MYTIME = 2022-01-02 03:04:05
In Jenkinsfile
stage('list') {
steps{
sh script:'ansible-playbook -i ./hosts.ini ./ping_playbook.yml -e time=\\"$TIME\\" --limit ${ENV}', label: "ping test"
}
}
Jenkins will run it as
ansible-playbook -i ./lzhxjp-test-update/lzhxjp/hosts.ini .ping_playbook.yml -e 'time="2022-01-02' '11:22:33"' --limit sdktest
How do I put it so that Jenkins interpret it as -e time="2022-01-02 11:22:33"
Inside single-quoted strings no variables are expanded by Jenkins (Groovy to be exact).
Single-quoted strings are plain java.lang.String and don’t support interpolation.
https://groovy-lang.org/syntax.html#all-strings
Thus it is passed directly into the shell and therefore already subject to shell quoting, you can remove the \\ before the " and then the shell the will expand "$TIME" correctly.
I have checked many suggestions regarding the issue, but didn't find a suitable solution.
Initially my pipeline code was working fine but with a Jenkins security warning The following steps that have been detected may have insecure interpolation of sensitive variables. sh: [pass, user]
My code is as below:
withCredentials([usernamePassword(credentialsId: 'Jenkins-Nexus-tech', passwordVariable: 'pass', usernameVariable: 'user')]){
sh "wget --user=$user --password=$pass 'https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war' "
}
Warning:
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [pass, user]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
To avoid the Jenkins security warning as suggested in https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#interpolation-of-sensitive-environment-variables , I have modified my code as below. But with different permutation and combination of single and double quotes it is not working. It is throwing Bad substitution error:
withCredentials([usernamePassword(credentialsId: 'Jenkins-Nexus-tech', passwordVariable: 'pass', usernameVariable: 'user')]){
sh 'wget --user=$user --password=$pass \"https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war\" '
}
I have tried with 3 single quotes. with this I am not getting the warning but it is unable to substitute the variables in the wget URL:
withCredentials([usernamePassword(credentialsId: 'Jenkins-Nexus-tech', passwordVariable: 'pass', usernameVariable: 'user')]){
sh '''wget --user=$user --password=$pass 'https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war' '''
}
Error is as below as it can't substitute the values of the variable
[Pipeline] sh
+ wget --user=**** --password=**** https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war
HTTP request sent, awaiting response... 404 com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war
ERROR 404: com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war.
It would be really great if someone can assist here.
The suggestion outlined in the page you mention suggests you do not use Groovy string interpolation to pass sensitive data to sh steps. The suggestion is to let the sh script instead get the values from the environment.
However, in your case you still need the value held in the params.AuthorizationServerVersion in your sh step.
Groovy string interpolation is only used when the string is within double quotes. String interpolation is turned off when the string is in single quotes.
When a string is in double quotes, you can escape $s using a backslash. This will prevent inserting the contents of a variable into the string when you don't want this to occur.
What you can do, then, is use double quotes, escape the $s on the variables you want to read from the environment, and leave the $s un-escaped on the variables you want to access directly from your Groovy code.
An example, using the withCredentials plugin:
withCredentials([string(credentialsId: 'mytoken', variable: 'THE_TOKEN')]) {
sh "some_command --that-uses \$THE_TOKEN --as-an-environment-variable --and-also-uses $params.A_PARAMETER"
}
Here, the token is injected into the environment as a secret, and is used in the sh script by escaping the $. This means the actual Unix shell (bash, zsh, or whatever) will get the literal $ and expand it as specified by that shell language.
The $ for $params.A_PARAMETER is not escaped, and is within double quotes, so Groovy will perform string interpolation and replace $params.A_PARAMETER with the value of the parameter at runtime as the string is constructed in memory, before the string is passed as an argument to the sh step.
In your specific case, this is what you can use (within your withCredentials block, of course):
sh "wget --user=\$user --password=\$pass 'https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war'"
Note that the $ on $user and $pass are escaped, but the $ on ${params.AuthorizationServerVersion} is not escaped.
I am using Choice param in my jenkins file to select environment as follows:
pipeline {
agent any
parameters {
choice(
name: 'ENVIRONMENT_URL',
choices: "https://beta1.xyz.com\nhttps://beta2.xyz.com\nhttps://beta3.xyz.com",
description: 'interesting stuff' )
}
in the Stage section, i have the following piece
stage('execute tests') {
steps {
script {
sh """URL=${ENVIRONMENT_URL} npm run e2e:tests"""
sh 'echo -e "\nTest Run Completed.\n"'
}
}
}
However, when i run the pipeline job by selecting the choice parameter i added, the following gets executed (the inserted choice param creates a line break):
+ URL=https://beta1.xyz.com
+ npm run e2e:tests
Using the variable is causing a line break and that's what is causing issue.
I have tried different ways to avoid line break. Tried using a variable but that didn't help. tried with different quotations, that didn't either.
What can i do to avoid line break ?
You can use the trim method on a String class type to remove trailing whitespace and newline:
sh "URL=${params.ENVIRONMENT_URL.trim()} npm run e2e:tests"
Note I also specified your parameter is in the params map and removed the triple quotes as those are for multiline string formatting.
Alternatively, you can specify the choices as an array instead of as a multiline string. The choices argument would then appear like:
choice(
name: 'ENVIRONMENT_URL',
choices: ['https://beta1.xyz.com', 'https://beta2.xyz.com', 'https://beta3.xyz.com'],
description: 'interesting stuff'
)
Either solution would fix your issue.
I have a Jenkinsfile which uses three tick marks surrounding a command to execute as in:
sh ''' command '''
We have no idea why three tick marks are required or what role they perform.
This syntax is seen in the Jenkinsfile doc set.
This has nothing at all to do with bash (in which triple-quotes have no special meaning at all), and everything to do with Groovy (the separate, non-bash interpreter that parses Jenkinsfiles).
In Groovy, but not in bash, strings must use triple-quotes to span multiple lines.
In the context of a sh directive in a Jenkinsfile, the content of your triple-quoted string is passed to a shell as a script to execute; however, the syntax is parsed by Groovy, so it's only Groovy that cares about the quotes themselves (as opposed to the quoted content).
Can you give more idea about what kind of command is it, is it a unix command or some script ?
The single quote and its variation like '''(3 ticks) as mentioned in question skip the variable expansion, and it could used to show what is being executed.
echo '''Updating JAVA_HOME variable :
export $JAVA_HOME="$NEW_JAVA_HOME" '''
However in your question, a command (some string) is enclosed between 3 ticks marks and sh tries to execute this command or script. One such example below
$ echo "echo hello" > /tmp/tesh.sh
$ sh '''/tmp/test.sh'''
hello