I'm trying to perform a shell script from a groovy function loaded by a jenkins-pipeline to retrieve a zip file from an external location. I am building the address out in the function and passing it into the shell script via $. But I am getting a syntax error and I'm not sure why.
I've tried escaping the $ but dont think thats the correct approach here and my code has been coverted from triple single quotes (''') to triple double (""") so I can pass the variable in.
def DownloadBaseLineFromNexus(groupID, artifactID){
//add code for this method
def nexusLink = "${GetNexusLink()}/${GetNexusProdRepo()}/${groupID}/${artifactID}/"
sh """
# retrieving all available version from release repo to versionFile.xml
curl ${nexusLink} | grep "<a href=.*</a>" | grep "http" | cut -d'>' -f3 |cut -d'/' -f1 > versionFile.xml
# creating array from versionFile.xml
fileItemString=$(cat versionFile.xml |tr "\n" " ")
fileItemArray=($fileItemString)
# Finding maximum of array element
maxValue=`printf "%d\n" "${fileItemArray[#]}" | sort -rn | head -1`
# Download latest version artifact from nexus
curl -o ${(artifactID)}.zip ${(nexusLink)}/${(artifactID)}-$maxValue.zip
# Unzip the tool
unzip ${(artifactID)}.zip
"""
}
the results I get are:
Script1.groovy: 28: illegal string body character after dollar sign;
solution: either escape a literal dollar sign "\$5" or bracket the value expression "${5}" # line 28, column 22.
curl "${nexusLink}" | grep "" | grep "http" | cut -d'>' -f3 |cut -d'/' -f1 > versionFile.xml
You have to add escape characters like below:-
curl ${nexusLink} | grep \"<a href=.*</a>\" | grep \"http\" | cut -d'>' -f3 |cut -d'/' -f1 > versionFile.xml
Related
I got .txt file with city names, each in separate line. Some of them are few words with one or multiple spaces or words connected with '-'. I need to create bash command which will echo those lines out. Currently I'm using cat piped with grep but I can't get both spaces and dash into one search and I had problems with checking for multiple spaces.
print lines with dash:
cat file.txt | grep ".*-.*"
print lines with spaces:
cat file.txt | grep ".*\s.*"
tho when I try to do:
cat file.txt | grep ".*\s+.*"
I get nothing.
Thanks for help
Something like that should work:
grep -E -- ' |\-' file.txt
Explanation:
-E: to interpret patterns as extended regular expressions
--: to signify the end of command options
' |\-': the line contains either a space or a dash
This does not directly address your question, but is too much to put in a comment.
You don't need the .* in your patterns. .* at the beginning or end of a pattern is useless, because it means "0 or more of any character" and so will always match.
These lines are all identical:
cat file.txt | grep ".*-.*"
cat file.txt | grep "-.*"
cat file.txt | grep "-"
Plus you don't need to cat and pipe:
grep "-" file.txt
When grep pattern matches, the default action is to print the whole line, so .* in all your patterns are redundant, you may delete them. Also, you don't have to use cat file | as you may specify the file to grep directly after pattern, i.e. grep 'pattern' file.txt.
Here are some more details:
grep ".*-.*" = grep -- "-" - returns any lines having a - char (-- singals the end of options, the next thing is the pattern)
grep ".*\s.*" = grep "\s" - matches and returns lines containing a whitespace char (only GNU grep)
grep ".*\s+.*" = grep "\s+" - returns line containing a whitespace followed with a literal + char (since you are using POSIX BRE regex here the unescaped + matches a literal plus symbol).
You want
grep "[[:space:]-]" file.txt
See the online demo:
#!/bin/bash
s='abc - def
ghi
jkl mno'
grep '[[:space:]-]' <<< "$s"
Output:
abc - def
jkl mno
The [[:space:]-] POSIX BRE and ERE (enabled with -E option) compliant pattern matches either any whitespace (with the [:space:] POSIX character class) or a hyphen.
Note that [\s-] won't work since \s inside a bracket expression is not treated as a regex escape sequence but as a mere \ or s.
After a lengthy pipe which ends with a grep, I correctly end up with a set of matching absolute paths/files and match string separated by a comma delimiter for each. I want to tag each file with its match string. Complicated also in that the path has spaces but there is none between the delimiter and the preceding and succeeding characters.
I need to be able to deal with an absolute path rather than just the filename within the directory. The match strings are space_free but the filename might not be:
So by way of example, the output of the pipe might look like:
pipe1 | pipe2 |
outputs
/Users/bloggs/Directory One/matched_file.doc,attributes_0001ABC
/Users/bloggs/Directory One/matched_file1.doc,attributeY_2
/Users/bloggs/Directory One/match_file_00x.doc,Attribute_00201
/Users/bloggs/Directory One/matching file 2.doc,attribute_0004
I want to tag each using something which will probably include:
tag --add "$attribute" "$file"
Where attribute refers to the match string eg "Attribute_00201"
Normally I'd just say eg:
tag --add Attribute_00201 /Users/bloggs/Directory\ One/match_file_00x.doc
At this point I am stuck how to parse each line ideally via another pipe and to deal with spaces correctly and execute the tag command. Grateful for any help
So I'm looking for a new pipe, pipe3 to execute or give me the correctly formatted tag command:
pipe1 | pipe2 | pipe3
delivers eg
tag --add Attribute_00201 /Users/bloggs/Directory\ One/match_file_00x.doc
etc
etc
This seems to work
| tee >(cut -f2 -d","| sed 's/^/tag --add /' > temp_out.txt) >(cut -d"," -f1 | sed -e 's/[[:space:]]/\\ /g' > temp_out1.txt) > /dev/null && paste -d' ' temp_out.txt temp_out1.txt > command.sh && chmod +x ./command.sh
New to scripting. I have only one line & one file. How do I capture summerfruit value (ie "mango") & pass it to another variable from the below line.
.. abc.dfe summer.fruit=mango summer.vegetable=potato projects.blah ...
If your grep supports Perl-compatible regular expressions (PCRE):
summerfruit=$(grep -Po 'summer\.fruit=\K[^ ]+' file)
The \K doesn't print the matched summer.fruit= and [^ ]+ matches one or more non-space characters after the =.
without PCRE:
summerfruit=$(grep -o 'summer\.fruit=[^ ]*' file | grep -o '[^=]*$')
With sed:
summerfruit=$(sed 's/.*summer\.fruit=\([^ ]*\).*/\1/' file)
With awk:
summerfruit=$(awk '{
for (i=1;i<=NF;i++)
if ($i ~ /^summer\.fruit=/){ sub(/^[^=]*=/,"",$i); print $i; exit }
}' file)
I have a set of many .json.gz files. In each file, there are entries such as this:
{"type":"e1","public":true, "login":"username1", "org":{"dict","of":"lots_of_things"}}
{"type":"e2","public":true, "login":"username2"}
No matter where in each nested dict "login" appears, I want to be able to detect it and take the username, only if the key "org" does not exist anywhere in the nested dict. I also want to count the number of times each username appears in the files.
My final output should be a file of dicts that looks like this:
{'username2: 1}
because of course username1 wouldn't be counted: the key "org" appears in its dict.
I'm looking for something like:
zgrep -Rv "org" . | zgrep -o 'login":"[^"]*"' /path/to/files/* | cut -d'"' -f3 | sort | uniq -c | sed '1i{
s/\s*\([0-9]*\)\s*\(.*\)/"\2": \1,/;$a}' > outputfile.txt
I'm not sure about this part:
zgrep -Rv "org" . |
The rest successfully creates the type of file I'm looking for. I'm just unsure about the order of operations here.
EDIT
I should have been more clear, I apologize. There are also often multiple instances of the key "login" per main dict object. For example (using "k" for any key that is not login and not org, and using "v" for a value):
{"k":"v","k":{"k":{"k":"v","login":"username1"},"k":"v"},"k":{"k":"v","login":"username2"}}
{"k":{"k":"v","k":"v"},"k":{"org":{"k":"v","k":v,"login":"username3"},"k":"v"},"k":{"k":"v","login":"username4"}}
{"k":{"k":"v"},"k":{"k":{"k":"v","login":"username1"},"login":"username2"}}
Since the key org appears in the second dict, I want to exclude usernames 3 and 4 from the dict I make and save to a file.
For example, I want this in a file:
{'username1': 2}
{'username2': 2}
AWK solution and replacing find -R with more reliable find:
find . -type f -name "*.json.gz" -print0 | xargs -0 zgrep -v -h '"org"' | awk '{ if ( match($0,/"login":"[^"]+"/) ) logins[substr($0,RSTART+8,RLENGTH-8)]++; } END { for ( i in logins ) print("{" i ":" logins[i] "}"); }'
Example output:
{"username2":1}
not grep but gnu sed job with script, your data in 'a'
i=
for e in $(sed -nE '/.*\borg\b.*/!s/.*"login":"(\w+)".*/{\1:}/p' a)
{
let i++;echo ${e/:/:$i}
}
use '>' at end to save in file
if better regex : 'pcregrep' installed, it does as well;
pcregrep -io '(?!.*\borg\b.*)(?<="login":")\w+(?=".*)' a
replace sed... script above, with a bit adjusted printout
This worked:
zgrep -v "org" *.json.gz | zgrep -o 'login":"[^"]*"' | cut -d'"' -f3 | sort | uniq -c | sed '1i{
s/\s*\([0-9]*\)\s*\(.*\)/"\2": \1,/;$a}' > usernames_2011.txt
I use egrep to output some lines with platform names:
XXX | egrep "i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$"
[30] i686-nptl-linux-gnu
[34] i686-w64-mingw32
[75] x86_64-unknown-linux-gnu
[77] x86_64-w64-mingw32
what I need is:
export PLATNUMS=30,34,75,77
How can I pipe the egrep command to sed / awk / bash script?
Try:
$ command | awk -F'[][ \t]+' '/i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$/{printf "%s%s",(f?",":"export PLATNUMS="),$2; f=1} END{print""}'
export PLATNUMS=30,34,75,77
How it works
-F'[][ \t]+'
Use any number of spaces, tabs, or [ or ] as field separators.
/i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$/{...}`
For the lines of interest, perform the commands in curly braces.
printf "%s%s",(f?",":"export PLATNUMS="),$2; f=1
For the lines of interest, print what we want.
The variable f marks whether this is the first line of interest.
END{print""}
After reading all lines, print a newline.
Creating a shell variable
export PLATNUMS=$(command | awk -F'[][ \t]+' '/i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$/{printf "%s%s",(f?",":""),$2; f=1} END{print""}')
For example, if the file input contains your data:
$ export PLATNUMS=$(awk -F'[][ \t]+' '/i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$/{printf "%s%s",(f?",":""),$2; f=1} END{print""}' input)
$ declare -p PLATNUMS
declare -x PLATNUMS="30,34,75,77"
For those who prefer their commands spread out over multiple lines:
export PLATNUMS=$(command | awk -F'[][ \t]+' '
/i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$/{
printf "%s%s",(f?",":""),$2
f=1
}
END{
print""
}
')
Perhaps this way, I can't try with your egrep.
export PLATNUMS=$(XXX | egrep "i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$" | sed ':A;s/\[\([[0-9]*\)].*/\1/;$bB;N;bA;:B;s/\n/,/g')
echo $PLATNUMS
How this work ?
Your egrep command return a multiline text
so sed read this text line by line this way
sed '
:A # label A
# here with your example
# on the first line the pattern space look like that
# [30] i686-nptl-linux-gnu
# on the second line the pattern space look like
# 30
# [34] i686-w64-mingw32
s/\[\([[0-9]*\)].*/\1/ # substitute all digit enclose by [] by only the digit
# on the first line the pattern space become
# 30
# on the second line the pattern space become
# 30
# 34
# and so on for each line
$bB # on the last line jump to B
N # get a newline in the pattern space
bA # It is not the last line so jump to A
:B # label B
# here we have read all the line
# the pattern space look like that without the #
# 30
# 34
# 75
# 77
s/\n/,/g' # subtitute all \n by a comma
# the pattern space become
# 30,34,75,77
# $(XXX | egrep .... | sed ...) return 30,34,75,77 in the variable PLATNUMS
# It is better not to use all capital letters in your variable name
With GNU sed and tr:
$ XXX | egrep "i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$" | sed -E 's,]\s+.+$,,g' | sed 's,^\[,,g' | tr '\n' ',' | sed -E 's,(^.+$),export PLATNUMS=\1,' | sed 's/,$//' && echo
I'm not sure what you want to achieve but you might want to automatically eval the output export:
$ eval $(XXX | egrep "i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$" | sed -E 's,]\s+.+$,,g' | sed 's,^\[,,g' | tr '\n' ',' | sed -E 's,(^.+$),export PLATNUMS=\1,' | sed 's/,$//' && echo)
$ echo $PLATNUMS
30,34,75,77
If you ever think you need grep+sed or 2 greps or 2 seds or any other combination then you should use 1 call to awk instead, and you never need grep or sed when you're using awk:
export PLATNUMS=$(XXX | awk -F'[][]' '/(i686-nptl-linux-gnu|i686-w64-mingw32|x86_64-unknown-linux-gnu|x86_64-w64-mingw32)$/{p=(p ? p "," : "") $2} END{print p}')
Btw in case it's useful, here's a couple of briefer regexps:
(i686-(nptl-linux-gnu|w64-mingw32)|x86_64-(unknown-linux-gnu|w64-mingw32))$
((i686-nptl|x86_64-unknown)-linux-gnu|(i686|x86_64)-w64-mingw32)$
and depending on your input data (since this will include combinations not provided by the above) you MIGHT only need:
(i686|x86_64)-(nptl|unknown|w64)-(linux-gnu|mingw32)$