I'm doing a simple substitution, and this works fine at the command line:
sed "s/pub Url =.*/pub Url = 'https:\/\/example.com:3207';/g" myfile.ts
I'm trying to run it within a Jenkinsfile, and like 40 builds later I cannot get the escape quoting right.
Pretty sure it will look something like this:
sh 'sed \\"s/pub Url =.*/pub Url = \\'https:\\\/\\\/example.com:3207\\';/g\\" myfile.ts'
Yet that results in the following error:
WorkflowScript: 4: unexpected char: '\' # line 4, column 49.
ub Url =.*/pub Url = \\'https:\\\/\\\/ex
I feel like I've tried dozens of variants but nothing is working.
Here is among the most common errors I'm getting: which points to escaping issue
sed: -e expression #1, char 1: unknown command: `"'
I really just need a pipeline expert that can likely see exactly what I'm doing wrong and know where to quote it.
As noted here: https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4 this is not uncommon to fight this type of stuff in the pipeline files and it seems like it's just trial and error.
Ok after much trial and error, this is working.
Looks like I had to use triple single quotes around the command. Good thing I don't need to interpolate!
sh '''sed \"s/pub Url =.*/pub Url = \\'https:\\/\\/example.com:3207\\';/g\" afile.txt'''
Hope this is helpful to someone in the future that's fighting this!
To add more on this... I had an issue with sed -i 's/\_/\//g' abc.txt command running fine in CLI but resulting in an unxpected char \ error in jenkins, so, i had to replace this:
sed -i 's/\_/\//g' abc.txt
with
sed -i "s/\\_/\\//g" abc.txt
To remove this error from jenkins.
Related
I am a Jenkins beginner.
Why does this command work?
sed -i -E s/'image: '(.*)${stack_name}-${service_name}:.*\$/'image: '\1${stack_name}-${service_name}:${version}/g
And why does the same command not work when it is included in a Jenkinsfile?
sh "sed -i -E s/'image: '(.*)${stack_name}-${service_name}:.*\$/'image: '\1${stack_name}-${service_name}:${version}/g"
The error is:
/opt/jenkins_data/workspace/secuview-front_master-Z2ADTSIGTSEJOG3UYRU4FPDUF5VZMB3SMQLEOUD46TUZG4POWKYQ#tmp/durable-a484faaf/script.sh: line 2: syntax error near unexpected token `('
Under the hood Jenkinsfiles are essentially Apache Groovy scripts, therefore string escaping rules for Groovy apply. When you have slashes they need to be escaped (e.g. \ -> \\) and when you're using double quotes using ${} literals actually get interpreted by the script instead of being passed to the shell step.
Try this instead:
sh 'sed -i -E s/\'image: \'\\(.*\\)${stack_name}-${service_name}:.*\\$/\'image: \'\\1${stack_name}-${service_name}:${version}/g'
I use Jenkins ver. 1.522 and I want to pass a long string with spaces and quotes as a parameter in the parameterized build section. The job only runs a python script.
My problem is that I can't find a way to escape my string so that jenkins passes it correctly to the script.
Assuming...
string: fixVersion in ("foo") AND issuetype in (Bug, Improvement) AND resolution = Fixed ORDER BY resolution ASC, assignee ASC, key DESC
variable name: bar
script name: coco.py
When I run the script in the terminal, everything is fine: python coco.py --option 'fixVersion in ("foo") AND issuetype in (Bug, Improvement) AND resolution = Fixed ORDER BY resolution ASC, assignee ASC, key DESC'
When I run the same script with jenkins using the parametrized build and try to escape the variable so it end up taken as one parameter by the py script it is oddly espacped by jenkins.
In my jenkins job I call the script: python coco.py --option \'${BAR}\'
and it ends up as:
python coco.py --option '"fixVersion' in '('\''foo'\'')' AND issuetype in '(Bug,' 'Improvement)' in '(Production,' 'Stage)' AND resolution = Fixed ORDER BY resolution ASC, assignee ASC, key 'DESC"'
I also tried \"${BAR}\", \"$BAR\",\'$BAR\'
What it the right way do acheive it?
Try
python coco.py --option "${BAR}"
Alternatively, if you need the single quotes surrounding everything
python coco.py --option \'"${BAR}"\'
In the cases you listed, bash will treat the spaces as delimiters. Putting the double quotes around a variable will preserve the whitespace in a string. Example
aString='foo bar'
for x in $aString; do echo $x; done
# foo
# bar
for x in "$aString"; do echo $x; done
# foo bar
I am using Jenkins v1.606 and ran into this same issue!
The issue that I saw passing user defined string params containing spaces into an execution shell would not properly format the string (only with a parameter that had 1 or more spaces). What you have to watch out for is reviewing the 'output' log. Jenkins will not properly display the string param value within the log.
Example (correct format for containing spaces):
docker exec -i container-base /bin/bash -c "cd /container/path/to/code/ && ./gradlew test_xml -P DISPLAY_NAME='${DISPLAY_NAME}' -P USERNAME='${USERNAME}' -P SERVER_NAME='${SERVER_NAME}'"
Jenkins Output of string (notice the string values format):
+ docker exec -i container-base /bin/bash -c 'cd /container/path/to/code/ && ./gradlew test_xml -P DISPLAY_NAME='\''VM10 USER D33PZ3R0'\'' -P USERNAME='\''d33pz3r0#stackoverflow.com'\'' -P SERVER_NAME='\''stackoverflow.com'\'''
Conclusion:
In my example, the literal command was encapsulated with <">, followed by surrounding the parameters with <'> to escape the literal cmd string and control the Jenkins string syntax. Remember not to just watch your Jenkins output log as it lead me wrong for an entire day while I fought with this! This should be the same for your issue as well, you do not need to escape with \' or other escape characters. Hope this helps!!
I have a external monitor job that I'm pushing the result of another job to it with curl and base on this link :
Monitoring external jobs
After I create the job I just need to run a curl command with the body encoded in HEX to the specified url and then a build will be created and the output will be added to it but what I get instead is part of my output in clear text and the rest in weird characters like so :
Started
Asking akamai to purge this urls:
http://xxx/sites/all/modules/custom/uk.png http://aaaaaasites/all/modules/custom/flags/jp.png
<html><head><title>401 Unauthorized</title> </h�VC��&�G����CV�WF��&��VC�������R&R��BWF��&��VBF�66W72F�B&W6�W&6S�����&�G�����F����F�RW&�F �6�V6�7FGW2�bF�R&WVW7B�2��F�RF��RF�v�B�2��6�Ɩ�r&6�w&�V�B��"F�6�V6�7FGW2�bF�RF�6�W#�v�F��rf�"���F�W&vRF��6O request please keep in mind this is an estimated time
Waiting for another 60 seconds
Asking akamai to purge this urls:
...
..
..
This is how I'm doing it :
export output=`cat msg.out|xxd -c 256 -ps`
curl -k -X POST -d "<run><log encoding=\"hexBinary\">$output</log><result>0</result> <duration>2000</duration></run>" https://$jenkinsuser:$jenkinspass#127.0.0.1/jenkins/job/akamai_purge_results/postBuildResult -H'.crumb:c775f3aa15464563456346e'
If I cat that file is all fine and even if I edit it with vi I can't see any problem with it.
Do you guys have any idea how to fix this ?
Could it be a problem with the hex encoding ? ( I tried hex/enc/dec pages with the result of xxd and they look fine)
Thanks.
I had the same issue, and stumbled across this: http://blog.markfeeney.com/2010/01/hexbinary-encoding.html
From that page, you can get the encoding you need via this command:
echo "Hello world" | hexdump -v -e '1/1 "%02x"'
48656c6c6f20776f726c640a
An excerpt from the explanation:
So what the hell is that? -v means don't suppress any duplicate data
in the output, and -e is the format string. hexdump's very particular
about the formatting of the -e argument; so careful with the quotes.
The 1/1 means for every 1 byte encountered in the input, apply the
following formatting pattern 1 time. Despite this sounding like the
default behaviour in the man page, the 1/1 is not optional. /1 also
works, but the 1/1 is very very slightly more readable, IMO. The
"%02x" is just a standard-issue printf-style format code.
So in your case, you would do this (removing 'export' in favor of inline variable)
OUTPUT=`cat msg.out | hexdump -v -e '1/1 "%02x"'` curl -k -X POST -d "<run><log encoding=\"hexBinary\">$OUTPUT</log><result>0</result> <duration>2000</duration></run>" https://$jenkinsuser:$jenkinspass#127.0.0.1/jenkins/job/akamai_purge_results/postBuildResult -H'.crumb:c775f3aa15464563456346e'
I am trying to parse my mvn verify output to only show lines with INFO tags. Please note that maven outputs line to stdout in real time and not by batch. I do not think that it is a problem with maven.
At first I tried to do it with grep:
$ mvn verify | grep INFO
but didn't seem to output lines in real time, as I understand grep buffers its lines before outputting, so I have to wait a few seconds between each flush and then I have tens of lines being printed at the same time, not very convenient. Then I thought I would try with sed.
According to this link, the following command:
sed -n '/PATTERN/p' file
// is equivalent to
grep PATTERN file
and according to this link, the -l option should force sed to flush its output buffer after every newline. So now I am using this command:
$ mvn verify | sed -ln -e '/INFO/p'
but I'm still getting the same result as before, I get a ton of output flushed every 30s or so and I don't know what I've done wrong. Can someone point me in the right direction please?
Try this, if your grep supports it:
mvn verify | grep --line-buffered INFO
If you're doing this in a terminal and still seeing buffered results, it would probably be something earlier than grep doing the buffering, but I'm not familiar with mvn. (And, yes, the -l option to sed should have done the same thing, so the problem may be upstream.)
try this line:
mvn verify | while read line; do echo $line|grep INFO; done
I found what was the problem, I was using a script to colorise maven output (see here) and in fact it was that script that was buffering the output down the pipe. I forgot about it as I was using it as an alias, I guess this is a good lesson, I won't alias as easily in the future. Anyway here is the fix, I changed -e to -le in the last line of the sed call:
mvn $# | sed -e "s/\(\[INFO\]\ \-.*\)/${TEXT_BLUE}${BOLD}\1/g" \
-e "s/\(\[INFO\]\ \[.*\)/${RESET_FORMATTING}${BOLD}\1${RESET_FORMATTING}/g" \
-e "s/\(\[INFO\]\ BUILD SUCCESSFUL\)/${BOLD}${TEXT_GREEN}\1${RESET_FORMATTING}/g" \
-e "s/\(\[WARNING\].*\)/${BOLD}${TEXT_YELLOW}\1${RESET_FORMATTING}/g" \
-e "s/\(\[ERROR\].*\)/${BOLD}${TEXT_RED}\1${RESET_FORMATTING}/g" \
-le "s/Tests run: \([^,]*\), Failures: \([^,]*\), Errors: \([^,]*\), Skipped: \([^,]*\)/${BOLD}${TEXT_GREEN}Tests run: \1${RESET_FORMATTING}, Failures: ${BOLD}${TEXT_RED}\2${RESET_FORMATTING}, Errors: ${BOLD}${TEXT_RED}\3${RESET_FORMATTING}, Skipped: ${BOLD}${TEXT_YELLOW}\4${RESET_FORMATTING}/g"
In effect this is telling sed to flush its output at every new line, which is what I wanted. I am sorry I didn't find another workaround that is more generic. I tried playing around with empty (see man page) and script but none of these solutions worked for me.
I tried to invoke java inside bash script on windows (Win XP) using cygwin.
However path to java.exe contain spaces.
only literaly putting in bash sometghing like this worked:
/cygdrive/c/Program\ Files/Java/jdk1.5.0_10/bin/java -cp "$TOOL_HOME" DateParse "$DATE" "$FORMAT"
My attemts to put java path to a variable failed:
export JAVA_EXE="/cygdrive/c/Program\ Files/Java/jdk1.5.0_10/bin/java"
$JAVA_EXE -cp "$TOOL_HOME" DateParse "$DATE" "$FORMAT"
also different combination with cygpath, quotes, brackets did not work. I am not finding the the right combination
Put quotes around $JAVA_EXE:
"$JAVA_EXE" -cp "$TOOL_HOME" DateParse "$DATE" "$FORMAT"
The problem is that every time a variable is expanded, its also broken into words at spaces, UNLESS you put quotes around it. So if you don't want things broken at spaces, you need quotes.
Another alternative is to always use short (DOS) names for things, which don't allow spaces. To see what the short name is, run
cygpath -d "$JAVA_EXE"
to convert that back to a unix-like cygwin path, use
cygpath -u $(cygpath -d "$JAVA_EXE")
thank you for your ideas. It worked in proper combination. The issue was that I was escaping space character and at the same time putting JAVA_EXE in quotes.
export JAVA_EXE="/cygdrive/c/Program Files/Java/jdk1.5.0_10/bin/java"
"$JAVA_EXE" -cp "$TOOL_HOME" DateParse "$DATE" "$FORMAT"
produce this effect:
line 30: /cygdrive/c/Program\ Files/Java/jdk1.5.0_10/bin/java: No such file or directory
on the other hand, converting to DOS 8.3 does not work neither:
cannot create short name of \\?\C:\Program\ Files\Java\jdk1.5.0_10
\bin\java
Finally, putting JAVA_EXE in quotes but without escaping space in path worked fine for me:
export JAVA_EXE="/cygdrive/c/Program Files/Java/jdk1.5.0_10/bin/java"
"$JAVA_EXE" -cp "$TOOL_HOME" DateParse "$DATE" "$FORMAT"