I am trying to execute ansible playbook in all EC2 AWS instances using Jenkinsfile function and assume-role.
But I am getting below error.
Obtained devops/JenkinsfileDynamic from git git#bitbucket.org:tui-uk-dev/cng-airflow-dags.git
Running in Durability level: MAX_SURVIVABILITY
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 33: illegal string body character after dollar sign;
solution: either escape a literal dollar sign "\$5" or bracket the value expression "${5}" # line 33, column 134.
SION_TOKEN=${AWS_SESSION_TOKEN} AWS_DEFA
^
Jenkinsfile:-
def Host_Verification2() {
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'cant_be_disclosed']]) {
sh '''
aws sts assume-role --role-arn "arn:aws:iam::12345678901:role/cant_role_jenkins" --role-session-name "connect" > assume-role-output.txt
export AWS_ACCESS_KEY_ID=`cat assume-role-output.txt | jq -c '.Credentials.AccessKeyId' | tr -d '"' | tr -d ' '`
export AWS_SECRET_ACCESS_KEY=`cat assume-role-output.txt | jq -c '.Credentials.SecretAccessKey' | tr -d '"' | tr -d ' '`
export AWS_SESSION_TOKEN=`cat assume-role-output.txt | jq -c '.Credentials.SessionToken' | tr -d '"' | tr -d ' '`
rm assume-role-output.txt
sshagent(credentials: ['tuiuki-cng-dev']) {
sh '''
cd acm/
sudo AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" AWS_SESSION_TOKEN="${AWS_SESSION_TOKEN}" inventory/ec2.py --list --refresh-cache
sudo AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" AWS_SESSION_TOKEN="${AWS_SESSION_TOKEN}" AWS_DEFAULT_REGION="eu-central-1" ansible-playbook -i inventory/ec2.py plays/emr/find.yml
'''
}
'''
}
}
Like the exception is saying:
solution: either escape a literal dollar sign "\$5" or bracket the value expression "${5}"
Try this:
sudo AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" ansible-playbook -i inventory/ec2.py plays/emr/findplaybooks.yml
sudo AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/ec2.py --limit "tag_Name_cluster" plays/emr/find.yml --private-key=${SSH_KEY} -u hadoop
Related
I'm trying to call the following funtion in Jenkins:
def openShiftInstall(char CLUSTER_URL, char TOKEN) {
sh '''
oc login --token='''TOKEN''' --server='''CLUSTER_URL'''
if helm history --max 1 $ARTIFACT_ID 2>/dev/null | grep FAILED | cut -f1 | grep -q 1; then
helm delete --purge $ARTIFACT_ID
fi
helm upgrade --install -f container-root/helm/values-devref.yaml --set image.version=$RELEASE_VERSION $ARTIFACT_ID container-root/helm --namespace ${NAMESPACE} --debug
'''
}
in this stage
environment {
NAMESPACE = 'test'
OC_a_DEV_TOKEN = 123
OC_b_DEV_TOKEN = 456
OC_a_DEV_CLUSTER_URL = 'https://api.ocp4-a.net:443'
OC_b_DEV_CLUSTER_URL = 'https://api.ocp4-b.net:443'
}
steps {
container('oc') {
script {
openShift.openShiftInstall("${OC_a_DEV_CLUSTER_URL}", "${OC_a_DEV_TOKEN}")
//openShift.openShiftInstall(${OC_b_DEV_CLUSTER_URL}, ${OC_b_DEV_TOKEN})
}
}
}
I'm getting this error:
java.lang.NoSuchMethodError: No such DSL method 'openShiftInstall' found among steps
Somebody can help me to understand what I'm doing here wrong?
Maybe is there a better solution to loop through multiple cluster installation with different parameters? Right now I'm just trying to run the same function twice, but even one is not working with the parameters.
You seems to be accepting a Char in the function openShiftInstall but you are passing a String. Change the method signature like below.
def openShiftInstall(def CLUSTER_URL, def TOKEN) {
sh '''
oc login --token='''TOKEN''' --server='''CLUSTER_URL'''
if helm history --max 1 $ARTIFACT_ID 2>/dev/null | grep FAILED | cut -f1 | grep -q 1; then
helm delete --purge $ARTIFACT_ID
fi
helm upgrade --install -f container-root/helm/values-devref.yaml --set image.version=$RELEASE_VERSION $ARTIFACT_ID container-root/helm --namespace ${NAMESPACE} --debug
'''
}
I changed to the following and it seems to working now.
def openShiftInstall(String CLUSTER_URL, String TOKEN) {
unstash 'lapc-docker-root'
sh '''
oc login --token=''' + TOKEN + ''' --server=''' + CLUSTER_URL + '''
if helm history --max 1 $ARTIFACT_ID 2>/dev/null | grep FAILED | cut -f1 | grep -q 1; then
helm delete --purge $ARTIFACT_ID
fi
helm upgrade --install -f container-root/helm/values-devref.yaml --set image.version=$RELEASE_VERSION $ARTIFACT_ID container-root/helm --namespace ${NAMESPACE} --debug
'''
}
The string parameter in jenkins dest_cmd works fine and gets passed and read by ansible as below:
Jenkins pipeline script:
ansible-playbook -i /web/runcmd/allmwhosts.hosts /web/runcmd/copyfiles.yml -e dest_user=$dest_user -e '{ dest_cmd: $dest_cmd }' --tags validate"
The above works fine for all dest_cmd parameters however, it fails when the user enters single quotes ' as you can see below:
[Pipeline] sh
+ ansible-playbook -i /web/runcmd/allmwhosts.hosts /web/runcmd/copyfiles.yml -e dest_user=wluser -e '{ dest_cmd: arp `hostname` | cut -d' ' -f4 }' --tags validate
usage: ansible-playbook [-h] [--version] [-v] [-k]
[--private-key PRIVATE_KEY_FILE] [-u REMOTE_USER]
[-c CONNECTION] [-T TIMEOUT]
[--ssh-common-args SSH_COMMON_ARGS]
[--sftp-extra-args SFTP_EXTRA_ARGS]
[--scp-extra-args SCP_EXTRA_ARGS]
[--ssh-extra-args SSH_EXTRA_ARGS] [--force-handlers]
Can you please suggest how to resolve this issue?
The problem is the use of quotes within the single-quoted string:
'{ dest_cmd: arp `hostname` | cut -d' ' -f4 }'
problem quotes ^ ^
Try using double quotes:
'{ dest_cmd: arp `hostname` | cut -d" " -f4 }'
I have a script that works fine in sh on a linux host as well as inside an alpine container. But when I try executing that using docker exec <containerID> sh -c "<script>" it misbehaves. The script's function is to output stuff similar to ps.
systick=$(getconf CLK_TCK); for c in /proc/*/cmdline; do d=$(dirname $c); name=$(grep Name: $d/status); pid=$(basename $d); uid=$(grep Uid: $d/status); uid=$(echo ${uid#Uid:} | xargs); uid=${uid%% *}; user=$(grep :$uid:[0-9] /etc/passwd); user=${user%%:*}; cmdline=$(cat $c|xargs -0 echo); starttime=$(($(awk '{print $22}' $d/stat) / systick)); uptime=$(awk '{print int($1)}' /proc/uptime); elapsed=$(($uptime-$starttime)); echo $pid $user $elapsed $cmdline; done
EDIT: sh -c "<script>" has the same behavior.
You are not able to run this script from docker exec because the variables will be interpolated before they sent to the container (i.e., you are going to get values from your local machine, not from within the container).
In order to run it as you wish, you need to replace $ with \$ for every occurrence of $ in your script.
What might work better is to put your script into a file, then map the file to a location within the container, using -v (i.e., -v script.sh:/path/to/script.sh), and call the script via docker exec /path/to/script.sh
Part 1: A Working Answer
A Working One-Liner (Quoted For Use By Docker)
getProcessDataDef='shellQuoteWordsDef='"'"'shellQuoteWords() { sq="'"'"'"'"'"'"'"'"'"; dq='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'; for arg; do printf "'"'"'"'"'"'"'"'"'%s'"'"'"'"'"'"'"'"' " "$(printf '"'"'"'"'"'"'"'"'%s\n'"'"'"'"'"'"'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"; done; printf '"'"'"'"'"'"'"'"'\n'"'"'"'"'"'"'"'"'; }'"'"'; shellQuoteNullSeparatedStream() { xargs -0 sh -c "${shellQuoteWordsDef};"'"'"' shellQuoteWords "$#"'"'"' _; }; getProcessData() { systick=$(getconf CLK_TCK); for c in /proc/*/cmdline; do d=${c%/*}; pid=${d##*/}; name=$(awk '"'"'/^Name:/ { print $2 }'"'"' <"$d"/status); uid=$(awk '"'"'/^Uid:/ { print $2 }'"'"' <"$d"/status); pwent=$(getent passwd "$uid"); user=${pwent%%:*}; cmdline=$(shellQuoteNullSeparatedStream <"$c"); starttime=$(awk -v systick="$systick" '"'"'{print int($22 / systick)}'"'"' "$d"/stat); uptime=$(awk '"'"'{print int($1)}'"'"' /proc/uptime); elapsed=$((uptime-starttime)); echo "$pid $user $elapsed $cmdline"; done; }; getProcessData'
sh -c "$getProcessDataDef" # or docker exec <container> sh -c "$getProcessDataDef"
A Working One-Liner (Before Quoting/Escaping)
shellQuoteWordsDef='shellQuoteWords() { sq="'"'"'"; dq='"'"'"'"'"'; for arg; do printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"; done; printf '"'"'\n'"'"'; }'; shellQuoteNullSeparatedStream() { xargs -0 sh -c "${shellQuoteWordsDef};"' shellQuoteWords "$#"' _; }; getProcessData() { systick=$(getconf CLK_TCK); for c in /proc/*/cmdline; do d=${c%/*}; pid=${d##*/}; name=$(awk '/^Name:/ { print $2 }' <"$d"/status); uid=$(awk '/^Uid:/ { print $2 }' <"$d"/status); pwent=$(getent passwd "$uid"); user=${pwent%%:*}; cmdline=$(shellQuoteNullSeparatedStream <"$c"); starttime=$(awk -v systick="$systick" '{print int($22 / systick)}' "$d"/stat); uptime=$(awk '{print int($1)}' /proc/uptime); elapsed=$((uptime-starttime)); echo "$pid $user $elapsed $cmdline"; done; }; getProcessData "$#"
What Went Into That One-Liner
shellQuoteWordsDef='shellQuoteWords() { sq="'"'"'"; dq='"'"'"'"'"'; for arg; do printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"; done; printf '"'"'\n'"'"'; }'
shellQuoteNullSeparatedStream() {
xargs -0 sh -c "${shellQuoteWordsDef};"' shellQuoteWords "$#"' _
}
getProcessData() {
systick=$(getconf CLK_TCK)
for c in /proc/*/cmdline; do
d=${c%/*}; pid=${d##*/}
name=$(awk '/^Name:/ { print $2 }' <"$d"/status)
uid=$(awk '/^Uid:/ { print $2 }' <"$d"/status)
pwent=$(getent passwd "$uid")
user=${pwent%%:*}
cmdline=$(shellQuoteNullSeparatedStream <"$c")
starttime=$(awk -v systick="$systick" '{print int($22 / systick)}' "$d"/stat)
uptime=$(awk '{print int($1)}' /proc/uptime)
elapsed=$((uptime-starttime))
echo "$pid $user $elapsed $cmdline"
done
}
What Went Into The Shell-Quoting Helper Used By That One-Liner
To allow easier reading and editing, the function stringified above looks like:
# This is the function we're including in our code passed to xargs in-band above:
shellQuoteWords() {
sq="'"; dq='"'
for arg; do
printf "'%s' " "$(printf '%s\n' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
done
printf '\n'
}
Part 2: How That Answer Was Created
Python has an excellent shlex.quote() function (or pipes.quote() in Python 2) that can be used to generate a shell-quoted version of a string. In this context, that can be used as follows:
Python 3.7.6 (default, Feb 27 2020, 15:15:00)
[Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> s = r'''
... shellQuoteWords() {
... sq="'"; dq='"'
... for arg; do
... printf "'%s' " "$(printf '%s\n' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
... done
... printf '\n'
... }
... '''
>>> import shlex
>>> print(shlex.quote(s))
'
shellQuoteWords() {
sq="'"'"'"; dq='"'"'"'"'"'
for arg; do
printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
done
printf '"'"'\n'"'"'
}
'
That result is itself a perfectly valid string in shell. That is to say, one can run:
s='
shellQuoteWords() {
sq="'"'"'"; dq='"'"'"'"'"'
for arg; do
printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
done
printf '"'"'\n'"'"'
}
'
eval "$s"
shellQuoteWords "hello world" 'hello world' "hello 'world'" 'hello "world"'
...and get completely valid output.
The same process was followed to generate a string that evaluated to the definition of getProcessData.
Here is my groovy file
timestamps{
node('cf_slave'){
checkout scm
stage('Read the file') {
def PWD = pwd()
withEnv(["prj_option=${params.project}"]) {
def response =sh(returnStdout: true, script: 'sh \'jenkins/security/get_values.sh\'')
}
}
This is my get_values.sh file
echo "The project option is:" $prj_option
prj_name=$(echo "$prj_option" | tr '[:upper:]' '[:lower:]')
file_name="va_input_file_$prj_name.txt"
echo "The project option is:" $file_name
ls -la
chmod 775 jenkins/security/$file_name
ls -la
get_input_values() {
file=$1
IFS=''
while read line
do
if [ `echo ${line} | grep -E -c -w "NAME_SPACE" ` -gt 0 ]; then
NAME_SPACE=$(echo " ${line}" | cut -d'=' -f2)
echo "The name space value is $NAME_SPACE"
elif [ `echo ${line} | grep -E -c -w "IMAGE_NAMES" ` -gt 0 ]; then
values=$(echo " ${line}" | cut -d'=' -f2)
echo "THE DOCKERIMAGES are $DOCKER_IMAGES_NAMES"
else
echo "Please provide input for namespace and docker images to be scanned by VA_TOOl"
fi
done < ${file}
}
images=$(get_input_values ${file_name})
so here my text file is under jenkins/security folder of gitrepo but unfortunately I am getting this error:
16:05:28 + sh jenkins/security/get_values.sh
16:05:28 jenkins/security/get_values.sh: 16: jenkins/security/get_values.sh: cannot open va_input_file_icp.txt: No such file```
Unfortunately, there is a ticket for this in Jenkins (https://issues.jenkins-ci.org/browse/JENKINS-51245) which was closed as a duplicate of this ticket: (https://issues.jenkins-ci.org/browse/JENKINS-27413)
JENKINS-27413 was raised in 2015, and is still open. The File Parameter appears not to work in Jenkins Pipeline. It does however work when used in a Freestyle project. While not ideal, I would recommend changing your job to be a Freestyle project if that's feasible.
I have a file 'README.txt' which contains the line -
"version": "1.0.0-alpha-test.7"
Using a Jenkins Pipeline, I want to replace this line with
"version": "1.0.0-alpha-test.{BUILD_NUMBER}"
The following sed command works when I try it on a linux cluster
sed -i -E "s#(\"version\"[ ]*:[ ]*\".+alpha-test\.)[0-9]+\"#\1${BUILD_NUMBER}#g" README.txt
The same command does not work using a Jenkins Pipeline.
Tried with the following query but it doesn't work -
sh """
sed -i -E "s|([\"]version[\"][ ]*:[ ]*[\"].+alpha-test\\.)[0-9]+\"|\1${BUILD_NUMBER}|g" README.txt
cat README.txt
"""
/home/jenkins/workspace/test/test-pipeline#tmp/durable-eb774fcf/script.sh:
3:
/home/jenkins/workspace/test/test-pipeline#tmp/durable-eb774fcf/script.sh:
Syntax error: ")" unexpected
Best to use perl command instead...
script{
old_version = (sh (returnStdout: true, script:'''old_version=`cat version.cfg |grep VERSION=|cut -d "=" -f2`
echo $old_version''')).toString().trim()
sh """
if [ "$old_version" != $new_version ]; then
perl -pi -e "s,$old_version,$new_version,g" version.cfg
##-- git push operation --##
fi
"""
}