I have written a function to record http code after the application is up to verify if response code is appropriate. This is on windows agent hosted on Azure VM.
def result = bat (returnStdout: true,
script: "curl --output /dev/null --silent --write-out '\\n%{http_code} http://localhost:8080")
echo "HTTP Code: ${result}"
So as I run the pipeline I get the following response in console log
**F:\jenkin_workspace\workspace\curl-test-pipeline>curl --write-out '\n//localhost:8080**
I tried various methods like using double quotes, single quotes and even without quotes but none helps.
Please suggest where am I wrong here?
Related
I'm coding this function in Jenkins to query Artifactory:
def curlDockerArtifact(URL, registryName, moduleName, tag, token) {
def controlURI = "${URL}/artifactory/api/storage/${registryName}/${moduleName}/${tag}"
def result = sh(script: """
curl -I -H \'Authorization: Bearer $token\' \
https://$controlURI -o /dev/null -w \'%{http_code}\' -s
""", returnStdout: true)
}
But I get this warning which I'm tying to avoid.
Warning: A secret was passed to "sh" using Groovy string interpolation, which is insecure.
I tried using single quotes but the variables don't get correctly interpreted from Groovy. Any idea how to fix/refactor the code?
You have to keep double quotes like you do, but you need to escape the $ sign for the token. Like this :
curl -I -H \'Authorization: Bearer \$token\'
The groovy will not interpolate the variable and the correct value will be passed on the shell level.
More informations : https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#interpolation-of-sensitive-environment-variables
I'm running a curl command in my Jenkinsfile.
post {
success {
script {
sh '''
|SCORE=+1
|GERRIT_COMMENT="$(cat <<-EOL
|Sonar result was: SUCCESS
|Report: ${Jenkins_URL}/job/${JOB_NAME}/${BUILD_NUMBER}/artifact/report1.txt
|EOL
|)"
|curl -s -u ${apiToken}: ${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children | json_pp -json_opt pretty,canonical > report1.txt
|echo "Voting unsuccessful"
'''.stripMargin().stripIndent()
archiveArtifacts artifacts: 'report1.txt', fingerprint: true
echo 'I Succeeded'
}
}
But I get the error
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/bin/json_pp
I can't use jq as it's not installed and installing it isn't an option.
The curl command works fine on my terminal but is failing in my Jenkins pipeline.
Also, when I do this instead, it works.
post {
success {
script {
sh '''
|SCORE=+1
|GERRIT_COMMENT="$(cat <<-EOL
|Sonar result was: SUCCESS
|Report: ${Jenkins_URL}/job/${JOB_NAME}/${BUILD_NUMBER}/artifact/report1.txt
|EOL
|)"
|echo "Voting unsuccessful"
'''.stripMargin().stripIndent()
sh """
curl -s -u ${apiToken}: '${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children' | json_pp -json_opt pretty,canonical > report1.txt
"""
archiveArtifacts artifacts: 'report1.txt', fingerprint: true
echo 'I Succeeded'
}
}
But it throws a warning in the console output.
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [apiToken]
What am I doing wrong, please?
In a Jenkins pipeline, how would you properly pass a JSON response using curl into a file?
I recommend to not use shell scripts whenever it is possible. Shell scripts are not cross platform and require installing additional tools (e.g. curl).
In your case the curl call could be replaced by the httpRequest step.
First let's replace the curl call and saves the result in a componentTree.json file:
httpRequest(
url: "${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children",
authorization: 'id-of-credentials-which-was-used-to-create-the-apiToken-variable',
outputFile: 'componentTree.json'
)
You want to format the JSON data in a human-readable format, so let's use the readJSON and writeJSON steps:
def json = readJSON(file: 'componentTree.json')
writeJSON(json: json, file: 'report1.txt', pretty: 4)
Now the report1.txt file contains JSON formatted with indent 4.
The componentTree.json file is written and read only once, so let's decrease the number of the IO operations:
def response = httpRequest(
url: "${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children",
authorization: 'id-of-credentials-which-was-used-to-create-the-apiToken-variable'
)
def json = readJSON(text: response.content)
writeJSON(json: json, file: 'report1.txt', pretty: 4)
About the warning:
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [apiToken]
Secrets never should be interpolated because they may contain special characters which could be interpreted. Example:
my secret: My' https://example.org; cat /etc/passwd; echo \
command: curl -u '${password}' https://private.server/path/file.txt
After the interpolation the following command is called:
curl -u 'My' https://example.org; cat /etc/passwd; echo \' https://private.server/path/file.txt
There are two options to fix it:
if apiToken is an environment variable:
sh "curl -s -u \$apiToken: '${Sonar_URL}/api/measures/component..."
if apiToken is a Groovy variable:
withEnv(["CREDENTIALS=${apiToken}"]) {
sh "curl -s -u \$CREDENTIALS: '${Sonar_URL}/api/measures/component..."
}
In both cases the dollar sign ($) is escaped before the credentials which means that shell script will resolve it (it will be taken from environment variables).
I have checked many suggestions regarding the issue, but didn't find a suitable solution.
Initially my pipeline code was working fine but with a Jenkins security warning The following steps that have been detected may have insecure interpolation of sensitive variables. sh: [pass, user]
My code is as below:
withCredentials([usernamePassword(credentialsId: 'Jenkins-Nexus-tech', passwordVariable: 'pass', usernameVariable: 'user')]){
sh "wget --user=$user --password=$pass 'https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war' "
}
Warning:
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [pass, user]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
To avoid the Jenkins security warning as suggested in https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#interpolation-of-sensitive-environment-variables , I have modified my code as below. But with different permutation and combination of single and double quotes it is not working. It is throwing Bad substitution error:
withCredentials([usernamePassword(credentialsId: 'Jenkins-Nexus-tech', passwordVariable: 'pass', usernameVariable: 'user')]){
sh 'wget --user=$user --password=$pass \"https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war\" '
}
I have tried with 3 single quotes. with this I am not getting the warning but it is unable to substitute the variables in the wget URL:
withCredentials([usernamePassword(credentialsId: 'Jenkins-Nexus-tech', passwordVariable: 'pass', usernameVariable: 'user')]){
sh '''wget --user=$user --password=$pass 'https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war' '''
}
Error is as below as it can't substitute the values of the variable
[Pipeline] sh
+ wget --user=**** --password=**** https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war
HTTP request sent, awaiting response... 404 com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war
ERROR 404: com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war.
It would be really great if someone can assist here.
The suggestion outlined in the page you mention suggests you do not use Groovy string interpolation to pass sensitive data to sh steps. The suggestion is to let the sh script instead get the values from the environment.
However, in your case you still need the value held in the params.AuthorizationServerVersion in your sh step.
Groovy string interpolation is only used when the string is within double quotes. String interpolation is turned off when the string is in single quotes.
When a string is in double quotes, you can escape $s using a backslash. This will prevent inserting the contents of a variable into the string when you don't want this to occur.
What you can do, then, is use double quotes, escape the $s on the variables you want to read from the environment, and leave the $s un-escaped on the variables you want to access directly from your Groovy code.
An example, using the withCredentials plugin:
withCredentials([string(credentialsId: 'mytoken', variable: 'THE_TOKEN')]) {
sh "some_command --that-uses \$THE_TOKEN --as-an-environment-variable --and-also-uses $params.A_PARAMETER"
}
Here, the token is injected into the environment as a secret, and is used in the sh script by escaping the $. This means the actual Unix shell (bash, zsh, or whatever) will get the literal $ and expand it as specified by that shell language.
The $ for $params.A_PARAMETER is not escaped, and is within double quotes, so Groovy will perform string interpolation and replace $params.A_PARAMETER with the value of the parameter at runtime as the string is constructed in memory, before the string is passed as an argument to the sh step.
In your specific case, this is what you can use (within your withCredentials block, of course):
sh "wget --user=\$user --password=\$pass 'https://nexus.mycompany.io/repository/updated-maven-releases/com/mycompany/project/authorization-server/${params.AuthorizationServerVersion}/authorization-server-${params.AuthorizationServerVersion}.war'"
Note that the $ on $user and $pass are escaped, but the $ on ${params.AuthorizationServerVersion} is not escaped.
I have a Jenkins job MY_REMOTE_JOB on which I enabled Triger Builds Remotely to run it with Authentication Token.
I would like to trigger MY_REMOTE_JOB in another job (CALLING_JOB) with an input parameter $MYPARAMETER that is already set (in CALLING_JOB). I tried this command:
curl -k -u JENKINS_USER_JOHN:123TOKEN123 https://myserver.com:8443/jenkins/job/MY_REMOTE_JOB/buildWithParameters?token=123TOKEN123&INPUT_PARAMETER=$MYPARAMETER
The trigger / call of a job MY_REMOTE_JOB is actually successful, but the parameter ($MYPARAMETER) that should be passed to that job is somehow left behind, so the job gets executed without a parameter.
How should I modify my curl so it will pass the parameter to the MY_REMOTE_JOB ?
I am running on Jenkins 1.609.3
At the first glance I could say you are missing -X flag for Posting data
I had similar issues, I referred man page for curl and found -F flag to be useful as this emulates a filled-in form, please also see to that parameter names and their order are matching exactly with the parameter names and its order on Jenkins Build With Parameters Page/Form
For instance let us assume the order of your Parameter Section looks in the following Format in a Descriptive Pipeline Syntax
pipeline {
parameters {
string(name: "PARAMETER1", defaultValue: "", description: "Enter a Value")
string(name: "PARAMETER2", defaultValue: "", description: "Enter a Value")
string(name: "PARAMETER3", defaultValue: "", description: "Enter a Value")
string(name: "PARAMETER4", defaultValue: "", description: "Enter a Value")
string(name: "PARAMETER5", defaultValue: "", description: "Enter a Value")
}
}
the order of the parameters in the curl command should be similar
Based on the format of your curl command below
curl -k -u JENKINS_USER_JOHN:123TOKEN123 https://myserver.com:8443/jenkins/job/MY_REMOTE_JOB/buildWithParameters?token=123TOKEN123&INPUT_PARAMETER=$MYPARAMETER
I would recommend try the following format
curl -k -X POST -u JENKINS_USER_JOHN:123TOKEN123 "https://myserver.com:8443/jenkins/job/MY_REMOTE_JOB/buildWithParameters" -F PARAMETER1=${PARAMETER1_VALUE} -F PARAMETER2=${PARAMETER2_VALUE} -F PARAMETER3=${PARAMETER3_VALUE} -F PARAMETER34=${PARAMETER4_VALUE} -F PARAMETER5=${PARAMETER5_VALUE}
Here is the link for Jenkins Documentation
If you also intend to use CSRF Protection with proper authentication follow the Jenkins CSRF , with this being enabled you could use the value in the curl command with -H flag
so with CSRF your curl command will be with the following format
curl -k -X POST -u JENKINS_USER_JOHN:123TOKEN123 -H ${YOUR_CRUMB_VALUE} "https://myserver.com:8443/jenkins/job/MY_REMOTE_JOB/buildWithParameters" -F PARAMETER1=${PARAMETER1_VALUE} -F PARAMETER2=${PARAMETER2_VALUE} -F PARAMETER3=${PARAMETER3_VALUE} -F PARAMETER4=${PARAMETER4_VALUE} -F PARAMETER5=${PARAMETER5_VALUE}
Finally also find some other answers related to your question Here
Hope the above recommendation solves your issue, Good Luck
I am trying to escape the '#' in the below code for a Jenkins build:
sh "curl --dH "#$Tom" http://google.com"
How do I escape it?
Edit: If I use a \ in front of the # as displayed below:
sh "curl --dH "\#$Tom" http://google.com"
I get another error, stating unexpected character "\".
Try this. no need to escape # character instead of that you need to escape double quotes and $ mark
sh "curl --dH \"#\${Tom}\" http://google.com"
Update: if Tom is a variable,It can be inject into string like this ${Tom}
jenkins pipeline syntax is groovy so you can try them with this online groovy ide
https://www.jdoodle.com/execute-groovy-online