Ansible can't send json variable in curl call - jenkins

I'm trying to send a json variable ( {{parameter_var}} ) to a Jenkins job from Ansible using a curl call. Here is the task I'm using:
- name: Call Jenkins Job
shell: curl -H "crumb:{{ response.json.crumb }}" '{{ jenkinshost }}/job/{{ jenkinsjob }}/buildWithParameters?token={{ jenkinstoken }}' --data-urlencode json='{"parameter": [{"name":"parameter", "value":\""{{ parameter_var }}"\"}]}'
The error I'm getting is:
Syntax Error while loading YAML.\n mapping values are not allowed in this context
Ansible says the issue is:
YAML thought it was looking for the start of a\nhash/dictionary and was confused to see a second "{".
It seems that instead of accepting {{ parameter_var }} as a variable, it's trying to read it as just a value of "{{ parameter_var }}". I tried a few ways of adding or escaping quotes, but can't seem to figure out what Ansible/YAML is looking for. Am I just adding my quotes wrong, or can I just not send a variable using this method?
I printed out my {{ parameter_var }}. There's nothing unusual about it, so I don't think that's the issue:
{
"msg": {
"msg": "All items completed",
"changed": false,
"results": [
{
"content_length": "142",
...
} ...
}
The goal I'm trying to accomplish is to set parameter "parameter" in Jenkins to the value of {{ parameter_var }}
Here's where I found the syntax for sending the json to Jenkins: https://wiki.jenkins.io/display/JENKINS/Remote+access+API

There are a lot of things that need addressing in your question:
you should not be shell-ing out to curl when there is a uri: module in ansible designed for that purpose
related to that, you left off --fail from your curl, so a non-200 response will cause the shell: to exit 0 which is almost certainly not what you intended
you cannot have a : character followed by whitespace in a yaml document because that's how yaml defines keys in a dict
you should not try and build up JSON using jinja2 templates, rather build up the structure in a dict or list and feed it through | to_json to ensure correct quoting
However, if you insist on that syntax specifically, you can fix it with some light editing:
- name: Call Jenkins Job
shell: >-
curl --fail -H "crumb:{{ response.json.crumb }}"
'{{ jenkinshost }}/job/{{ jenkinsjob }}/buildWithParameters?token={{ jenkinstoken }}'
--data-urlencode json={{ json_data | to_json | quote }}
vars:
json_data:
parameter:
- name: "parameter"
value: '{{ parameter_var }}'

Related

CURL response from Jenkinsfile, http_code problematic due to escape chars

I have written a function to record http code after the application is up to verify if response code is appropriate. This is on windows agent hosted on Azure VM.
def result = bat (returnStdout: true,
script: "curl --output /dev/null --silent --write-out '\\n%{http_code} http://localhost:8080")
echo "HTTP Code: ${result}"
So as I run the pipeline I get the following response in console log
**F:\jenkin_workspace\workspace\curl-test-pipeline>curl --write-out '\n//localhost:8080**
I tried various methods like using double quotes, single quotes and even without quotes but none helps.
Please suggest where am I wrong here?

How to read environment variables in env section of github action workflow

I'm trying to set a env variable based on another env variable in a github workflow. I've tried a couple of syntax options but none seem to work
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
workflow_dispatch:
env:
BASE_VERSION: 1.0.0
FULL_VERSION: ${BASE_VERSION}-${{ github.run_number }}-${{ github.ref_name }}
jobs:
The example for BASE_VERSION above just keeps ${BASE_VERSION} as a string
$BASE_VERSION also just keeps $BASE_VERSION as a string
${{ env.BASE_VERSION }}-blabla just fails with syntax error
Is this doable?
The output I want is "1.0.0-1-master" for example
Is this doable?
It does not seem like a supported behaviour at the moment.
The docs on env mentions that
variables in the env map cannot be defined in terms of other variables in the map.
Do it like this:
- name: Set docker image env var
run: |
echo "DOCKER_IMAGE=${ARTIFACTORY_URL}/${IMAGE_NAME}:${GITHUB_REF##*/}.${{github.run_number}}" >> $GITHUB_ENV
- run: |
echo ${{ env.DOCKER_IMAGE }}
Outputs
artifactory-host/some-project/some-repo/image-name:branch.number

Ansible: How to debug mutiple tasks, commands or iterations?

In a YAML playbook, I need to run multiple commands through each element in an array. I am using with_items to iterate, but I am getting syntax errors when I try to blend debug into the tasks.
Everything works when I remove the debug module, but than I can't see whats happening. Example provided below.
Produces Syntax Error:
- name: Iterate through array and display results
shell: "run command 1 on {{ item }}"
register: command1
debug:
msg: "Command 1 || {{ command1.stdout_lines}}"
shell: "run command 2 on {{ item }}"
register: command2
debug:
msg: "Command 2 || {{ command2.stdout_lines}}"
with_items: "{{ array_name }}"
Everything seems to be working fine when I remove the debugs, but I need to see what each command produces because it has important info.
Is there some way to print/debug the results while keeping it inside the with_items iterations?
Edit:
I also tried a few more things but they also gave me a syntax error
Also Tried 1:
- name: Iterate through array and display results
- shell: "run command 1 on {{ item }}"
register: command1
- debug:
msg: "Command 1 || {{ command1.stdout_lines}}"
- shell: "run command 2 on {{ item }}"
register: command2
- debug:
msg: "Command 2 || {{ command2.stdout_lines}}"
with_items: "{{ array_name }}"
Also Tried 2:
- name: Iterate through array and display results
- shell: "run command 1 on {{ item }}"
register: command1
- debug:
msg: "Command 1 || {{ command1.stdout_lines}}"
- shell: "run command 2 on {{ item }}"
register: command2
- debug:
msg: "Command 2 || {{ command2.stdout_lines}}"
with_items: "{{ array_name }}"
As the comments of β.εηοιτ.βε already stated
In Ansible, you can only have one module per item in the list (tasks is a list of tasks, each being an Ansible module) ... you have to have a new item in the list (a dash -), in front of all the debug and shell tasks.
This means,
integrate debug into iterations
I try to blend debug into the tasks
you can't do debugging in that way since the debug_module is for
Print statements during execution
only. However, according your question
Is there some way to print/debug the results while keeping it inside the with_items iterations?
it seems for me that you are more interested in Debugging tasks, play, role or block itself. So for this
Ansible offers a task debugger so you can fix errors during execution instead of editing your playbook and running it again to see if your change worked.
For usage you have to Enabling the debugger with the debugger keyword and may have a look into the Examples, Resolving errors in the debugger and Available debug commands.
Further questions here on SO are
How to debug Ansible issues?
Does Ansible have any feature to allow for debugging/pausing?
I was able to solve the issue, Ansible does not allow iterations through multiple tasks. I had to put the tasks in a separate yml playbook and use include_tasks: to call the playbook.

How To Capture and Format Json Response in Jenkins Pipeline

I'm running a curl command in my Jenkinsfile.
post {
success {
script {
sh '''
|SCORE=+1
|GERRIT_COMMENT="$(cat <<-EOL
|Sonar result was: SUCCESS
|Report: ${Jenkins_URL}/job/${JOB_NAME}/${BUILD_NUMBER}/artifact/report1.txt
|EOL
|)"
|curl -s -u ${apiToken}: ${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children | json_pp -json_opt pretty,canonical > report1.txt
|echo "Voting unsuccessful"
'''.stripMargin().stripIndent()
archiveArtifacts artifacts: 'report1.txt', fingerprint: true
echo 'I Succeeded'
}
}
But I get the error
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/bin/json_pp
I can't use jq as it's not installed and installing it isn't an option.
The curl command works fine on my terminal but is failing in my Jenkins pipeline.
Also, when I do this instead, it works.
post {
success {
script {
sh '''
|SCORE=+1
|GERRIT_COMMENT="$(cat <<-EOL
|Sonar result was: SUCCESS
|Report: ${Jenkins_URL}/job/${JOB_NAME}/${BUILD_NUMBER}/artifact/report1.txt
|EOL
|)"
|echo "Voting unsuccessful"
'''.stripMargin().stripIndent()
sh """
curl -s -u ${apiToken}: '${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children' | json_pp -json_opt pretty,canonical > report1.txt
"""
archiveArtifacts artifacts: 'report1.txt', fingerprint: true
echo 'I Succeeded'
}
}
But it throws a warning in the console output.
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [apiToken]
What am I doing wrong, please?
In a Jenkins pipeline, how would you properly pass a JSON response using curl into a file?
I recommend to not use shell scripts whenever it is possible. Shell scripts are not cross platform and require installing additional tools (e.g. curl).
In your case the curl call could be replaced by the httpRequest step.
First let's replace the curl call and saves the result in a componentTree.json file:
httpRequest(
url: "${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children",
authorization: 'id-of-credentials-which-was-used-to-create-the-apiToken-variable',
outputFile: 'componentTree.json'
)
You want to format the JSON data in a human-readable format, so let's use the readJSON and writeJSON steps:
def json = readJSON(file: 'componentTree.json')
writeJSON(json: json, file: 'report1.txt', pretty: 4)
Now the report1.txt file contains JSON formatted with indent 4.
The componentTree.json file is written and read only once, so let's decrease the number of the IO operations:
def response = httpRequest(
url: "${Sonar_URL}/api/measures/component_tree?ps=100&s=qualifier,name&component=sonarqube&metricKeys=ncloc,bugs,vulnerabilities,code_smells,security_hotspots,coverage,duplicated_lines_density&strategy=children",
authorization: 'id-of-credentials-which-was-used-to-create-the-apiToken-variable'
)
def json = readJSON(text: response.content)
writeJSON(json: json, file: 'report1.txt', pretty: 4)
About the warning:
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [apiToken]
Secrets never should be interpolated because they may contain special characters which could be interpreted. Example:
my secret: My' https://example.org; cat /etc/passwd; echo \
command: curl -u '${password}' https://private.server/path/file.txt
After the interpolation the following command is called:
curl -u 'My' https://example.org; cat /etc/passwd; echo \' https://private.server/path/file.txt
There are two options to fix it:
if apiToken is an environment variable:
sh "curl -s -u \$apiToken: '${Sonar_URL}/api/measures/component..."
if apiToken is a Groovy variable:
withEnv(["CREDENTIALS=${apiToken}"]) {
sh "curl -s -u \$CREDENTIALS: '${Sonar_URL}/api/measures/component..."
}
In both cases the dollar sign ($) is escaped before the credentials which means that shell script will resolve it (it will be taken from environment variables).

How to pass a parameter to Jenkins using curl and Authentication Token

I have a Jenkins job MY_REMOTE_JOB on which I enabled Triger Builds Remotely to run it with Authentication Token.
I would like to trigger MY_REMOTE_JOB in another job (CALLING_JOB) with an input parameter $MYPARAMETER that is already set (in CALLING_JOB). I tried this command:
curl -k -u JENKINS_USER_JOHN:123TOKEN123 https://myserver.com:8443/jenkins/job/MY_REMOTE_JOB/buildWithParameters?token=123TOKEN123&INPUT_PARAMETER=$MYPARAMETER
The trigger / call of a job MY_REMOTE_JOB is actually successful, but the parameter ($MYPARAMETER) that should be passed to that job is somehow left behind, so the job gets executed without a parameter.
How should I modify my curl so it will pass the parameter to the MY_REMOTE_JOB ?
I am running on Jenkins 1.609.3
At the first glance I could say you are missing -X flag for Posting data
I had similar issues, I referred man page for curl and found -F flag to be useful as this emulates a filled-in form, please also see to that parameter names and their order are matching exactly with the parameter names and its order on Jenkins Build With Parameters Page/Form
For instance let us assume the order of your Parameter Section looks in the following Format in a Descriptive Pipeline Syntax
pipeline {
parameters {
string(name: "PARAMETER1", defaultValue: "", description: "Enter a Value")
string(name: "PARAMETER2", defaultValue: "", description: "Enter a Value")
string(name: "PARAMETER3", defaultValue: "", description: "Enter a Value")
string(name: "PARAMETER4", defaultValue: "", description: "Enter a Value")
string(name: "PARAMETER5", defaultValue: "", description: "Enter a Value")
}
}
the order of the parameters in the curl command should be similar
Based on the format of your curl command below
curl -k -u JENKINS_USER_JOHN:123TOKEN123 https://myserver.com:8443/jenkins/job/MY_REMOTE_JOB/buildWithParameters?token=123TOKEN123&INPUT_PARAMETER=$MYPARAMETER
I would recommend try the following format
curl -k -X POST -u JENKINS_USER_JOHN:123TOKEN123 "https://myserver.com:8443/jenkins/job/MY_REMOTE_JOB/buildWithParameters" -F PARAMETER1=${PARAMETER1_VALUE} -F PARAMETER2=${PARAMETER2_VALUE} -F PARAMETER3=${PARAMETER3_VALUE} -F PARAMETER34=${PARAMETER4_VALUE} -F PARAMETER5=${PARAMETER5_VALUE}
Here is the link for Jenkins Documentation
If you also intend to use CSRF Protection with proper authentication follow the Jenkins CSRF , with this being enabled you could use the value in the curl command with -H flag
so with CSRF your curl command will be with the following format
curl -k -X POST -u JENKINS_USER_JOHN:123TOKEN123 -H ${YOUR_CRUMB_VALUE} "https://myserver.com:8443/jenkins/job/MY_REMOTE_JOB/buildWithParameters" -F PARAMETER1=${PARAMETER1_VALUE} -F PARAMETER2=${PARAMETER2_VALUE} -F PARAMETER3=${PARAMETER3_VALUE} -F PARAMETER4=${PARAMETER4_VALUE} -F PARAMETER5=${PARAMETER5_VALUE}
Finally also find some other answers related to your question Here
Hope the above recommendation solves your issue, Good Luck

Resources