The string parameter in jenkins dest_cmd works fine and gets passed and read by ansible as below:
Jenkins pipeline script:
ansible-playbook -i /web/runcmd/allmwhosts.hosts /web/runcmd/copyfiles.yml -e dest_user=$dest_user -e '{ dest_cmd: $dest_cmd }' --tags validate"
The above works fine for all dest_cmd parameters however, it fails when the user enters single quotes ' as you can see below:
[Pipeline] sh
+ ansible-playbook -i /web/runcmd/allmwhosts.hosts /web/runcmd/copyfiles.yml -e dest_user=wluser -e '{ dest_cmd: arp `hostname` | cut -d' ' -f4 }' --tags validate
usage: ansible-playbook [-h] [--version] [-v] [-k]
[--private-key PRIVATE_KEY_FILE] [-u REMOTE_USER]
[-c CONNECTION] [-T TIMEOUT]
[--ssh-common-args SSH_COMMON_ARGS]
[--sftp-extra-args SFTP_EXTRA_ARGS]
[--scp-extra-args SCP_EXTRA_ARGS]
[--ssh-extra-args SSH_EXTRA_ARGS] [--force-handlers]
Can you please suggest how to resolve this issue?
The problem is the use of quotes within the single-quoted string:
'{ dest_cmd: arp `hostname` | cut -d' ' -f4 }'
problem quotes ^ ^
Try using double quotes:
'{ dest_cmd: arp `hostname` | cut -d" " -f4 }'
Related
I have a script that works fine in sh on a linux host as well as inside an alpine container. But when I try executing that using docker exec <containerID> sh -c "<script>" it misbehaves. The script's function is to output stuff similar to ps.
systick=$(getconf CLK_TCK); for c in /proc/*/cmdline; do d=$(dirname $c); name=$(grep Name: $d/status); pid=$(basename $d); uid=$(grep Uid: $d/status); uid=$(echo ${uid#Uid:} | xargs); uid=${uid%% *}; user=$(grep :$uid:[0-9] /etc/passwd); user=${user%%:*}; cmdline=$(cat $c|xargs -0 echo); starttime=$(($(awk '{print $22}' $d/stat) / systick)); uptime=$(awk '{print int($1)}' /proc/uptime); elapsed=$(($uptime-$starttime)); echo $pid $user $elapsed $cmdline; done
EDIT: sh -c "<script>" has the same behavior.
You are not able to run this script from docker exec because the variables will be interpolated before they sent to the container (i.e., you are going to get values from your local machine, not from within the container).
In order to run it as you wish, you need to replace $ with \$ for every occurrence of $ in your script.
What might work better is to put your script into a file, then map the file to a location within the container, using -v (i.e., -v script.sh:/path/to/script.sh), and call the script via docker exec /path/to/script.sh
Part 1: A Working Answer
A Working One-Liner (Quoted For Use By Docker)
getProcessDataDef='shellQuoteWordsDef='"'"'shellQuoteWords() { sq="'"'"'"'"'"'"'"'"'"; dq='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'; for arg; do printf "'"'"'"'"'"'"'"'"'%s'"'"'"'"'"'"'"'"' " "$(printf '"'"'"'"'"'"'"'"'%s\n'"'"'"'"'"'"'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"; done; printf '"'"'"'"'"'"'"'"'\n'"'"'"'"'"'"'"'"'; }'"'"'; shellQuoteNullSeparatedStream() { xargs -0 sh -c "${shellQuoteWordsDef};"'"'"' shellQuoteWords "$#"'"'"' _; }; getProcessData() { systick=$(getconf CLK_TCK); for c in /proc/*/cmdline; do d=${c%/*}; pid=${d##*/}; name=$(awk '"'"'/^Name:/ { print $2 }'"'"' <"$d"/status); uid=$(awk '"'"'/^Uid:/ { print $2 }'"'"' <"$d"/status); pwent=$(getent passwd "$uid"); user=${pwent%%:*}; cmdline=$(shellQuoteNullSeparatedStream <"$c"); starttime=$(awk -v systick="$systick" '"'"'{print int($22 / systick)}'"'"' "$d"/stat); uptime=$(awk '"'"'{print int($1)}'"'"' /proc/uptime); elapsed=$((uptime-starttime)); echo "$pid $user $elapsed $cmdline"; done; }; getProcessData'
sh -c "$getProcessDataDef" # or docker exec <container> sh -c "$getProcessDataDef"
A Working One-Liner (Before Quoting/Escaping)
shellQuoteWordsDef='shellQuoteWords() { sq="'"'"'"; dq='"'"'"'"'"'; for arg; do printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"; done; printf '"'"'\n'"'"'; }'; shellQuoteNullSeparatedStream() { xargs -0 sh -c "${shellQuoteWordsDef};"' shellQuoteWords "$#"' _; }; getProcessData() { systick=$(getconf CLK_TCK); for c in /proc/*/cmdline; do d=${c%/*}; pid=${d##*/}; name=$(awk '/^Name:/ { print $2 }' <"$d"/status); uid=$(awk '/^Uid:/ { print $2 }' <"$d"/status); pwent=$(getent passwd "$uid"); user=${pwent%%:*}; cmdline=$(shellQuoteNullSeparatedStream <"$c"); starttime=$(awk -v systick="$systick" '{print int($22 / systick)}' "$d"/stat); uptime=$(awk '{print int($1)}' /proc/uptime); elapsed=$((uptime-starttime)); echo "$pid $user $elapsed $cmdline"; done; }; getProcessData "$#"
What Went Into That One-Liner
shellQuoteWordsDef='shellQuoteWords() { sq="'"'"'"; dq='"'"'"'"'"'; for arg; do printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"; done; printf '"'"'\n'"'"'; }'
shellQuoteNullSeparatedStream() {
xargs -0 sh -c "${shellQuoteWordsDef};"' shellQuoteWords "$#"' _
}
getProcessData() {
systick=$(getconf CLK_TCK)
for c in /proc/*/cmdline; do
d=${c%/*}; pid=${d##*/}
name=$(awk '/^Name:/ { print $2 }' <"$d"/status)
uid=$(awk '/^Uid:/ { print $2 }' <"$d"/status)
pwent=$(getent passwd "$uid")
user=${pwent%%:*}
cmdline=$(shellQuoteNullSeparatedStream <"$c")
starttime=$(awk -v systick="$systick" '{print int($22 / systick)}' "$d"/stat)
uptime=$(awk '{print int($1)}' /proc/uptime)
elapsed=$((uptime-starttime))
echo "$pid $user $elapsed $cmdline"
done
}
What Went Into The Shell-Quoting Helper Used By That One-Liner
To allow easier reading and editing, the function stringified above looks like:
# This is the function we're including in our code passed to xargs in-band above:
shellQuoteWords() {
sq="'"; dq='"'
for arg; do
printf "'%s' " "$(printf '%s\n' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
done
printf '\n'
}
Part 2: How That Answer Was Created
Python has an excellent shlex.quote() function (or pipes.quote() in Python 2) that can be used to generate a shell-quoted version of a string. In this context, that can be used as follows:
Python 3.7.6 (default, Feb 27 2020, 15:15:00)
[Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> s = r'''
... shellQuoteWords() {
... sq="'"; dq='"'
... for arg; do
... printf "'%s' " "$(printf '%s\n' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
... done
... printf '\n'
... }
... '''
>>> import shlex
>>> print(shlex.quote(s))
'
shellQuoteWords() {
sq="'"'"'"; dq='"'"'"'"'"'
for arg; do
printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
done
printf '"'"'\n'"'"'
}
'
That result is itself a perfectly valid string in shell. That is to say, one can run:
s='
shellQuoteWords() {
sq="'"'"'"; dq='"'"'"'"'"'
for arg; do
printf "'"'"'%s'"'"' " "$(printf '"'"'%s\n'"'"' "$arg" | sed -e "s#${sq}#${sq}${dq}${sq}${dq}${sq}#g")"
done
printf '"'"'\n'"'"'
}
'
eval "$s"
shellQuoteWords "hello world" 'hello world' "hello 'world'" 'hello "world"'
...and get completely valid output.
The same process was followed to generate a string that evaluated to the definition of getProcessData.
I am trying to execute ansible playbook in all EC2 AWS instances using Jenkinsfile function and assume-role.
But I am getting below error.
Obtained devops/JenkinsfileDynamic from git git#bitbucket.org:tui-uk-dev/cng-airflow-dags.git
Running in Durability level: MAX_SURVIVABILITY
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 33: illegal string body character after dollar sign;
solution: either escape a literal dollar sign "\$5" or bracket the value expression "${5}" # line 33, column 134.
SION_TOKEN=${AWS_SESSION_TOKEN} AWS_DEFA
^
Jenkinsfile:-
def Host_Verification2() {
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'cant_be_disclosed']]) {
sh '''
aws sts assume-role --role-arn "arn:aws:iam::12345678901:role/cant_role_jenkins" --role-session-name "connect" > assume-role-output.txt
export AWS_ACCESS_KEY_ID=`cat assume-role-output.txt | jq -c '.Credentials.AccessKeyId' | tr -d '"' | tr -d ' '`
export AWS_SECRET_ACCESS_KEY=`cat assume-role-output.txt | jq -c '.Credentials.SecretAccessKey' | tr -d '"' | tr -d ' '`
export AWS_SESSION_TOKEN=`cat assume-role-output.txt | jq -c '.Credentials.SessionToken' | tr -d '"' | tr -d ' '`
rm assume-role-output.txt
sshagent(credentials: ['tuiuki-cng-dev']) {
sh '''
cd acm/
sudo AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" AWS_SESSION_TOKEN="${AWS_SESSION_TOKEN}" inventory/ec2.py --list --refresh-cache
sudo AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" AWS_SESSION_TOKEN="${AWS_SESSION_TOKEN}" AWS_DEFAULT_REGION="eu-central-1" ansible-playbook -i inventory/ec2.py plays/emr/find.yml
'''
}
'''
}
}
Like the exception is saying:
solution: either escape a literal dollar sign "\$5" or bracket the value expression "${5}"
Try this:
sudo AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" ansible-playbook -i inventory/ec2.py plays/emr/findplaybooks.yml
sudo AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/ec2.py --limit "tag_Name_cluster" plays/emr/find.yml --private-key=${SSH_KEY} -u hadoop
I have a file 'README.txt' which contains the line -
"version": "1.0.0-alpha-test.7"
Using a Jenkins Pipeline, I want to replace this line with
"version": "1.0.0-alpha-test.{BUILD_NUMBER}"
The following sed command works when I try it on a linux cluster
sed -i -E "s#(\"version\"[ ]*:[ ]*\".+alpha-test\.)[0-9]+\"#\1${BUILD_NUMBER}#g" README.txt
The same command does not work using a Jenkins Pipeline.
Tried with the following query but it doesn't work -
sh """
sed -i -E "s|([\"]version[\"][ ]*:[ ]*[\"].+alpha-test\\.)[0-9]+\"|\1${BUILD_NUMBER}|g" README.txt
cat README.txt
"""
/home/jenkins/workspace/test/test-pipeline#tmp/durable-eb774fcf/script.sh:
3:
/home/jenkins/workspace/test/test-pipeline#tmp/durable-eb774fcf/script.sh:
Syntax error: ")" unexpected
Best to use perl command instead...
script{
old_version = (sh (returnStdout: true, script:'''old_version=`cat version.cfg |grep VERSION=|cut -d "=" -f2`
echo $old_version''')).toString().trim()
sh """
if [ "$old_version" != $new_version ]; then
perl -pi -e "s,$old_version,$new_version,g" version.cfg
##-- git push operation --##
fi
"""
}
I'm trying to automate a JMX command using a tool called JMXterm.
The script looks like so:
#!/bin/bash
beanstemp="/tmp/jmx_beans"
read -r -p "Enter server name in FQDN " scrapername
bean1="BrokerName=localhost,Connection=Scraper_$scrapername,ConnectorName=openwire,Type=Connection"
echo beans -d $domain | $cmd > $beanstemp
idnum=$(grep "ID_$scrapername" $beanstemp | grep openwire | awk -F- '{print $2"-"$3}')
idname="$scrapername-$idnum"
bean2="BrokerName=localhost,Connection=ID_"$idname"2_0,ConnectorName=openwire,Type=Connection"
domain="org.apache.activemq"
server="scrapermq.sj.peer39.com:1099"
cmd="java -jar Downloads/jmxterm-1.0-alpha-4-uber.jar -l $server"
echo run stop -b $bean1 -d $domain | $cmd -l $server
echo run stop -b $bean2 -d $domain | $cmd -l $server
When I run the script I get the proper response for the first command but the second one fails.
Here's the output of sh -x:
itaig#iganot-lt:~$ sh -x jmx.sh
+ beanstemp=/tmp/jmx_beans
+ read -r -p Enter server name in FQDN scrapername
Enter server name in FQDN selvmnj27.domain.company.com
+ bean1=BrokerName=localhost,Connection=Scraper_selvmnj27.domain.company.com,ConnectorName=openwire,Type=Connection
+ echo beans -d
+
+ + grep openwire
grep ID_selvmnj27.domain.company.com+ awk -F- {print $2"-"$3}
/tmp/jmx_beans
+ idnum=
+ idname=selvmnj27.domain.company.com-
+ bean2=BrokerName=localhost,Connection=ID_selvmnj27.domain.company.com-2_0,ConnectorName=openwire,Type=Connection
+ domain=org.apache.activemq
+ server=scrapermq.sj.peer39.com:1099
+ cmd=java -jar Downloads/jmxterm-1.0-alpha-4-uber.jar -l scrapermq.sj.peer39.com:1099
+ echo+ run stop -b BrokerName=localhost,Connection=Scraper_selvmnj27.domain.company.com,ConnectorName=openwire,Type=Connectionjava -jar -d Downloads/jmxterm-1.0-alpha-4-uber.jar -l org.apache.activemq scrapermq.sj.peer39.com:1099 -l
scrapermq.sj.peer39.com:1099
Welcome to JMX terminal. Type "help" for available commands.
$>run stop -b BrokerName=localhost,Connection=Scraper_selvmnj27.domain.company.com,ConnectorName=openwire,Type=Connection -d org.apache.activemq
#calling operation stop of mbean org.apache.activemq:BrokerName=localhost,Connection=Scraper_selvmnj27.domain.company.com,ConnectorName=openwire,Type=Connection
#operation returns:
null
$>+ + echo runjava stop -b -jar Downloads/jmxterm-1.0-alpha-4-uber.jar BrokerName=localhost,Connection=ID_selvmnj27.domain.company.com-2_0,ConnectorName=openwire,Type=Connection -l scrapermq.sj.peer39.com:1099 -l -d scrapermq.sj.peer39.com:1099
org.apache.activemq
Welcome to JMX terminal. Type "help" for available commands.
$>run stop -b BrokerName=localhost,Connection=ID_selvmnj27.domain.company.com-2_0,ConnectorName=openwire,Type=Connection -d org.apache.activemq
#InstanceNotFoundException: org.apache.activemq:BrokerName=localhost,Connection=ID_selvmnj27.domain.company.com-2_0,ConnectorName=openwire,Type=Connection
$>itaig#iganot-lt:~$
"Operation returns: null" is the response I'm looking for.
The problem begins with this line:
echo beans -d $domain | $cmd > $beanstemp
When it runs the output of the command should be written to the $beanstemp file but the file stays empty.
When I run the command manually from terminal it works:
itaig#iganot-lt:~$ echo beans -d org.apache.activemq | java -jar Downloads/jmxterm-1.0-alpha-4-uber.jar -l scrapermq.domain.company.com:1099 > /tmp/jmx_beans
Welcome to JMX terminal. Type "help" for available commands.
$>beans -d org.apache.activemq
#domain = org.apache.activemq:
$>itaig#iganot-lt:~$ tail -5 /tmp/jmx_beans
org.apache.activemq:BrokerName=localhost,Type=Subscription,clientId=Scraper_selvmnj28.domain.company.com,consumerId=ID_selvmnj28.domain.company.com-45117-1431866382308-0_416_1_1,destinationName=SCRAPER_VIDEO_INPUT,destinationType=Queue,persistentMode=Non-Durable
org.apache.activemq:BrokerName=localhost,Type=Subscription,clientId=Scraper_selvmnj29.domain.company.com,consumerId=ID_selvmnj29.domain.company.com-52961-1431866382264-0_416_-1_1,destinationName=topic_//ActiveMQ.Advisory.TempQueue_topic_//ActiveMQ.Advisory.TempTopic,destinationType=Topic,persistentMode=Non-Durable
org.apache.activemq:BrokerName=localhost,Type=Subscription,clientId=Scraper_selvmnj29.domain.company.com,consumerId=ID_selvmnj29.domain.company.com-52961-1431866382264-0_416_1_1,destinationName=SCRAPER_VIDEO_INPUT,destinationType=Queue,persistentMode=Non-Durable
org.apache.activemq:BrokerName=localhost,Type=Subscription,clientId=Scraper_selvmnj30.domain.company.com,consumerId=ID_selvmnj30.domain.company.com-33036-1431866396456-0_416_-1_1,destinationName=topic_//ActiveMQ.Advisory.TempQueue_topic_//ActiveMQ.Advisory.TempTopic,destinationType=Topic,persistentMode=Non-Durable
org.apache.activemq:BrokerName=localhost,Type=Subscription,clientId=Scraper_selvmnj30.domain.company.com,consumerId=ID_selvmnj30.domain.company.com-33036-1431866396456-0_416_1_1,destinationName=SCRAPER_VIDEO_INPUT,destinationType=Queue,persistentMode=Non-Durable
itaig#iganot-lt:~$
So my question is: What am I doing wrong and why does the command work when running manually from terminal but not from the script?
Thanks in advance
Ok I found the problem.
Certain lines included variables which have not been declared in time.
After re-organizing the order of variable declaration the problem has been solved.
zgrep -i XXX XXX | grep -o "RID=[0-9|A-Z]*" |
uniq | cut -d "=" -f2 |
xargs -0 -I string echo "RequestID="string
My output is
RequestID=121212112
8127127128
8129129812
But my requirement is to have the request ID prefixed before all the output.
Any help is appreciated
I had a similar task and this worked for me. It might be what you are looking for:
zgrep -i XXX XXX | grep -o "RID=[0-9|A-Z]*" |
uniq | cut -d "=" -f2 |
xargs -I {} echo "RequestID="{}
Try -n option of xargs.
-n max-args
Use at most max-args arguments per command line. Fewer than max-args arguments will be used if the size (see the -s option)
is exceeded,
unless the -x option is given, in which case xargs will exit.
Example:
$ echo -e '1\n2' | xargs echo 'str ='
str = 1 2
$ echo -e '1\n2' | xargs -n 1 echo 'str ='
str = 1
str = 2