I would like to grep a specific word 'foo' inside specific files, then get the N lines around my match and show only the blocks that contain a second grep.
I found this but it doesn't really work...
find . | grep -E '.*?\.(c|asm|mac|inc)$' | \
xargs grep --color -C3 -rie 'foo' | \
xargs -n1 --delimiter='--' | grep --color -l 'bar'
For instance I have the file 'a':
a
b
c
d
bar
f
foo
g
h
i
j
bar
l
The file b:
a
bar
c
d
e
foo
g
h
i
j
k
I expect this for grep -c2 on both files because bar is contained in the -c2 range of foo. I do not get any match for ./bar because bar is not in the range -c2 of foo...
--
./foo- bar
./foo- f
./foo- **foo**
./foo- g
./foo- h
--
Any ideas?
You could do this pretty simply with a "while read line" loop:
find -regextype posix-extended -regex "./file[a-z]" | while read line; do grep -nHC2 "foo" $line | grep --color bar; done
Output:
./filea-5-bar
./filec-46-... host pwns.me [94.23.120.252]: 451 4.7.1 Local bar
configuration error ...
In this example, I created the following files:
filea - your example a
fileb - your example b
filec - some random exim log output with foo and bar tossed in 2 lines apart
filed - the same exim log output, but with foo and bar tossed in 3 lines apart
You could also pipe the output after done, to alter the format:
; done | sed 's/-([0-9]{1,6})-/: line: \1 ::: /'
Formatted output
./filea: line: 5 ::: bar
./filec: line: 46 ::: ... host pwns.me [94.23.120.252]: 451 4.7.1 Local bar configuration error ...
I think I only understand the first line of your question and this does what I think you mean!
#!/bin/bash
N=2
pattern1=a
pattern2=z
matchinglines=$(awk -v p="$pattern1" '$0~p{print NR}' file) # Generate array of matching line numbers
for x in ${matchinglines[#]}
do
((start=x-N))
[[ $start -lt 1 ]] && start=1 # Avoid passing negative line nmumbers to sed
((end=x+N))
echo DEBUG: Checking block between lines $start and $end
sed -ne "${start},${end}p" file | grep -q "$pattern2"
[[ $? -eq 0 ]] && sed -ne "${start},${end}p" file
done
You need to set pattern1 and pattern2 at the start of the script. It basically does some awk to build an array of the line numbers that match your first pattern. Then it loops through the array and sets the start and end range to +/-N either side of each matching line number. It then uses sed to extraact that block and passes it through grep to see if it contains pattern2 printing it if it does. It may not be the most efficient, but it is easy enough to understand and maintain.
It assumes your file is called file
pipe it twice
grep "[^foo\n]" | grep "\n{ntimes}foo\n{ntimes}"
Related
Question: I'd like to print a single line directly following a line that contains a matching pattern.
My version of sed will not take the following syntax (it bombs out on +1p) which would seem like a simple solution:
sed -n '/ABC/,+1p' infile
I assume awk would be better to do multiline processing, but I am not sure how to do it.
Never use the word "pattern" in this context as it is ambiguous. Always use "string" or "regexp" (or in shell "globbing pattern"), whichever it is you really mean. See How do I find the text that matches a pattern? for more about that.
The specific answer you want is:
awk 'f{print;f=0} /regexp/{f=1}' file
or specializing the more general solution of the Nth record after a regexp (idiom "c" below):
awk 'c&&!--c; /regexp/{c=1}' file
The following idioms describe how to select a range of records given a specific regexp to match:
a) Print all records from some regexp:
awk '/regexp/{f=1}f' file
b) Print all records after some regexp:
awk 'f;/regexp/{f=1}' file
c) Print the Nth record after some regexp:
awk 'c&&!--c;/regexp/{c=N}' file
d) Print every record except the Nth record after some regexp:
awk 'c&&!--c{next}/regexp/{c=N}1' file
e) Print the N records after some regexp:
awk 'c&&c--;/regexp/{c=N}' file
f) Print every record except the N records after some regexp:
awk 'c&&c--{next}/regexp/{c=N}1' file
g) Print the N records from some regexp:
awk '/regexp/{c=N}c&&c--' file
I changed the variable name from "f" for "found" to "c" for "count" where
appropriate as that's more expressive of what the variable actually IS.
f is short for found. Its a boolean flag that I'm setting to 1 (true) when I find a string matching the regular expression regexp in the input (/regexp/{f=1}). The other place you see f on its own in each script it's being tested as a condition and when true causes awk to execute its default action of printing the current record. So input records only get output after we see regexp and set f to 1/true.
c && c-- { foo } means "if c is non-zero then decrement it and if it's still non-zero then execute foo" so if c starts at 3 then it'll be decremented to 2 and then foo executed, and on the next input line c is now 2 so it'll be decremented to 1 and then foo executed again, and on the next input line c is now 1 so it'll be decremented to 0 but this time foo will not be executed because 0 is a false condition. We do c && c-- instead of just testing for c-- > 0 so we can't run into a case with a huge input file where c hits zero and continues getting decremented so often it wraps around and becomes positive again.
It's the line after that match that you're interesting in, right? In sed, that could be accomplished like so:
sed -n '/ABC/{n;p}' infile
Alternatively, grep's A option might be what you're looking for.
-A NUM, Print NUM lines of trailing context after matching lines.
For example, given the following input file:
foo
bar
baz
bash
bongo
You could use the following:
$ grep -A 1 "bar" file
bar
baz
$ sed -n '/bar/{n;p}' file
baz
I needed to print ALL lines after the pattern ( ok Ed, REGEX ), so I settled on this one:
sed -n '/pattern/,$p' # prints all lines after ( and including ) the pattern
But since I wanted to print all the lines AFTER ( and exclude the pattern )
sed -n '/pattern/,$p' | tail -n+2 # all lines after first occurrence of pattern
I suppose in your case you can add a head -1 at the end
sed -n '/pattern/,$p' | tail -n+2 | head -1 # prints line after pattern
And I really should include tlwhitec's comment in this answer (since their sed-strict approach is the more elegant than my suggestions):
sed '0,/pattern/d'
The above script deletes every line starting with the first and stopping with (and including) the line that matches the pattern. All lines after that are printed.
awk Version:
awk '/regexp/ { getline; print $0; }' filetosearch
If pattern match, copy next line into the pattern buffer, delete a return, then quit -- side effect is to print.
sed '/pattern/ { N; s/.*\n//; q }; d'
Actually sed -n '/pattern/{n;p}' filename will fail if the pattern match continuous lines:
$ seq 15 |sed -n '/1/{n;p}'
2
11
13
15
The expected answers should be:
2
11
12
13
14
15
My solution is:
$ sed -n -r 'x;/_/{x;p;x};x;/pattern/!s/.*//;/pattern/s/.*/_/;h' filename
For example:
$ seq 15 |sed -n -r 'x;/_/{x;p;x};x;/1/!s/.*//;/1/s/.*/_/;h'
2
11
12
13
14
15
Explains:
x;: at the beginning of each line from input, use x command to exchange the contents in pattern space & hold space.
/_/{x;p;x};: if pattern space, which is the hold space actually, contains _ (this is just a indicator indicating if last line matched the pattern or not), then use x to exchange the actual content of current line to pattern space, use p to print current line, and x to recover this operation.
x: recover the contents in pattern space and hold space.
/pattern/!s/.*//: if current line does NOT match pattern, which means we should NOT print the NEXT following line, then use s/.*// command to delete all contents in pattern space.
/pattern/s/.*/_/: if current line matches pattern, which means we should print the NEXT following line, then we need to set a indicator to tell sed to print NEXT line, so use s/.*/_/ to substitute all contents in pattern space to a _(the second command will use it to judge if last line matched the pattern or not).
h: overwrite the hold space with the contents in pattern space; then, the content in hold space is ^_$ which means current line matches the pattern, or ^$, which means current line does NOT match the pattern.
the fifth step and sixth step can NOT exchange, because after s/.*/_/, the pattern space can NOT match /pattern/, so the s/.*// MUST be executed!
This might work for you (GNU sed):
sed -n ':a;/regexp/{n;h;p;x;ba}' file
Use seds grep-like option -n and if the current line contains the required regexp replace the current line with the next, copy that line to the hold space (HS), print the line, swap the pattern space (PS) for the HS and repeat.
Piping some greps can do it (it runs in POSIX shell and under BusyBox):
cat my-file | grep -A1 my-regexp | grep -v -- '--' | grep -v my-regexp
-v will show non-matching lines
-- is printed by grep to separate each match, so we skip that too
If you just want the next line after a pattern, this sed command will work
sed -n -e '/pattern/{n;p;}'
-n supresses output (quiet mode);
-e denotes a sed command (not required in this case);
/pattern/ is a regex search for lines containing the literal combination of the characters pattern (Use /^pattern$/ for line consisting of only of “pattern”;
n replaces the pattern space with the next line;
p prints;
For example:
seq 10 | sed -n -e '/5/{n;p;}'
Note that the above command will print a single line after every line containing pattern. If you just want the first one use sed -n -e '/pattern/{n;p;q;}'. This is also more efficient as the whole file is not read.
This strictly sed command will print all lines after your pattern.
sed -n '/pattern/,${/pattern/!p;}
Formatted as a sed script this would be:
/pattern/,${
/pattern/!p
}
Here’s a short example:
seq 10 | sed -n '/5/,${/5/!p;}'
/pattern/,$ will select all the lines from pattern to the end of the file.
{} groups the next set of commands (c-like block command)
/pattern/!p; prints lines that doesn’t match pattern. Note that the ; is required in early versions, and some non-GNU, of sed. This turns the instruction into a exclusive range - sed ranges are normally inclusive for both start and end of the range.
To exclude the end of range you could do something like this:
sed -n '/pattern/,/endpattern/{/pattern/!{/endpattern/d;p;}}
/pattern/,/endpattern/{
/pattern/!{
/endpattern/d
p
}
}
/endpattern/d is deleted from the “pattern space” and the script restarts from the top, skipping the p command for that line.
Another pithy example:
seq 10 | sed -n '/5/,/8/{/5/!{/8/d;p}}'
If you have GNU sed you can add the debug switch:
seq 5 | sed -n --debug '/2/,/4/{/2/!{/4/d;p}}'
Output:
SED PROGRAM:
/2/,/4/ {
/2/! {
/4/ d
p
}
}
INPUT: 'STDIN' line 1
PATTERN: 1
COMMAND: /2/,/4/ {
COMMAND: }
END-OF-CYCLE:
INPUT: 'STDIN' line 2
PATTERN: 2
COMMAND: /2/,/4/ {
COMMAND: /2/! {
COMMAND: }
COMMAND: }
END-OF-CYCLE:
INPUT: 'STDIN' line 3
PATTERN: 3
COMMAND: /2/,/4/ {
COMMAND: /2/! {
COMMAND: /4/ d
COMMAND: p
3
COMMAND: }
COMMAND: }
END-OF-CYCLE:
INPUT: 'STDIN' line 4
PATTERN: 4
COMMAND: /2/,/4/ {
COMMAND: /2/! {
COMMAND: /4/ d
END-OF-CYCLE:
INPUT: 'STDIN' line 5
PATTERN: 5
COMMAND: /2/,/4/ {
COMMAND: }
END-OF-CYCLE:
I am trying to find which words happened in logfiles plus show the logfilename for anything that matches following pattern:
'BA10\|BA20\|BA21\|BA30\|BA31\|BA00'
so if file dummylogfile.log contains BA10002 I would like to get a result such as:
dummylogfile.log:BA10002
it is totally fine if the logfile shows up twice for duplicate matches.
the closest I got is:
for f in $(find . -name '*.err' -exec grep -l 'BA10\|BA20\|BA21\|BA30\|BA31\|BA00' {} \+);do printf $f;printf ':';grep -o 'BA10\|BA20\|BA21\|BA30\|BA31\|BA00' $f;done
but this gives things like:
./register-05-14-11-53-59_24154.err:BA10
BA10
./register_mdw_files_2020-05-14-11-54-32_24429.err:BA10
BA10
./process_tables.2020-05-18-11-18-09_11428.err:BA30
./status_load_2020-05-18-11-35-31_9185.err:BA30
so,
1) there are empty lines with only the second match and
2) the full match (e.g., BA10004) is not shown.
thanks for the help
There are a couple of options you can pass to grep:
-H: This will report the filename and the match
-o: only show the match, not the full line
-w: The match must represent a full word (string build from [A-Za-z0-9_])
If we look at your regex, you use BA01, this will match only BA01 which can appear anywhere in the text, also mid word. If you want the regex to match a full word, it should read BA01[[:alnum:]_]* which adds any sequence of word-constituent characters (equivalent to [A-Za-z0-9_]). You can test this with
$ echo "foo BA01234 barBA012" | grep -Ho "BA01"
(standard input):BA01
(standard input):BA01
$ echo "foo BA01234 barBA012" | grep -How "BA01"
$ echo "foo BA01234 barBA012" | grep -How "BA01[[:alnum:]_]*"
(standard input):BA01234
So your grep should look like
grep -How "\('BA10\|BA20\|BA21\|BA30\|BA31\|BA00'\)[[:alnum:]_]*" *.err
From your example it seems that all files are in one directory. So the following works right away:
grep -l 'BA10\|BA20\|BA21\|BA30\|BA31\|BA00' *.err
If the files are in different directories:
find . -name '*.err' -print | xargs -I {} grep 'BA10\|BA20\|BA21\|BA30\|BA31\|BA00' {} /dev/null
Explanation: the addition of /dev/null to the filename {} forces grep to report the matching filename
How can I do an exact match using grep -v?
For example: the following command
for i in 0 0.3 a; do echo $i | grep -v "0"; done
returns a. But I want it to return 0.3 a.
Using
grep -v "0\b"
is not working
for i in 0 0.3 a; do echo $i | grep -v "^0$"; done
You need to match the start and end of the string with ^ and $
So, we say "match the beginning of a line, the char 0 and then the end of the line.
$ for i in 0 0.3 a; do echo $i | grep -v "^0$"; done
0.3
a
The safest way for single-column entries is using awk. Normally, I would use grep with the -w flag, but since you want to exactly match an integer that could be part of a float, it is a bit more tricky. The <dot>-character makes it hard to use any of
grep -vw 0
grep -v '\b0\b'
grep -v '\<0\>'
The proposed solution also will only work on perfect lines, what if you have a lost space in front or after your zero. The line will fail. So the safest would be:
single column file:
awk '($1!="0")' file
multi-word file: (adopt the variable FS to fit your needs)
awk '{for(i=1;i<=NF;++i) if($i == "0") next}1' file
I use egrep to output some lines with platform names:
XXX | egrep "i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$"
[30] i686-nptl-linux-gnu
[34] i686-w64-mingw32
[75] x86_64-unknown-linux-gnu
[77] x86_64-w64-mingw32
what I need is:
export PLATNUMS=30,34,75,77
How can I pipe the egrep command to sed / awk / bash script?
Try:
$ command | awk -F'[][ \t]+' '/i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$/{printf "%s%s",(f?",":"export PLATNUMS="),$2; f=1} END{print""}'
export PLATNUMS=30,34,75,77
How it works
-F'[][ \t]+'
Use any number of spaces, tabs, or [ or ] as field separators.
/i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$/{...}`
For the lines of interest, perform the commands in curly braces.
printf "%s%s",(f?",":"export PLATNUMS="),$2; f=1
For the lines of interest, print what we want.
The variable f marks whether this is the first line of interest.
END{print""}
After reading all lines, print a newline.
Creating a shell variable
export PLATNUMS=$(command | awk -F'[][ \t]+' '/i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$/{printf "%s%s",(f?",":""),$2; f=1} END{print""}')
For example, if the file input contains your data:
$ export PLATNUMS=$(awk -F'[][ \t]+' '/i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$/{printf "%s%s",(f?",":""),$2; f=1} END{print""}' input)
$ declare -p PLATNUMS
declare -x PLATNUMS="30,34,75,77"
For those who prefer their commands spread out over multiple lines:
export PLATNUMS=$(command | awk -F'[][ \t]+' '
/i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$/{
printf "%s%s",(f?",":""),$2
f=1
}
END{
print""
}
')
Perhaps this way, I can't try with your egrep.
export PLATNUMS=$(XXX | egrep "i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$" | sed ':A;s/\[\([[0-9]*\)].*/\1/;$bB;N;bA;:B;s/\n/,/g')
echo $PLATNUMS
How this work ?
Your egrep command return a multiline text
so sed read this text line by line this way
sed '
:A # label A
# here with your example
# on the first line the pattern space look like that
# [30] i686-nptl-linux-gnu
# on the second line the pattern space look like
# 30
# [34] i686-w64-mingw32
s/\[\([[0-9]*\)].*/\1/ # substitute all digit enclose by [] by only the digit
# on the first line the pattern space become
# 30
# on the second line the pattern space become
# 30
# 34
# and so on for each line
$bB # on the last line jump to B
N # get a newline in the pattern space
bA # It is not the last line so jump to A
:B # label B
# here we have read all the line
# the pattern space look like that without the #
# 30
# 34
# 75
# 77
s/\n/,/g' # subtitute all \n by a comma
# the pattern space become
# 30,34,75,77
# $(XXX | egrep .... | sed ...) return 30,34,75,77 in the variable PLATNUMS
# It is better not to use all capital letters in your variable name
With GNU sed and tr:
$ XXX | egrep "i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$" | sed -E 's,]\s+.+$,,g' | sed 's,^\[,,g' | tr '\n' ',' | sed -E 's,(^.+$),export PLATNUMS=\1,' | sed 's/,$//' && echo
I'm not sure what you want to achieve but you might want to automatically eval the output export:
$ eval $(XXX | egrep "i686-nptl-linux-gnu$|i686-w64-mingw32$|x86_64-unknown-linux-gnu$|x86_64-w64-mingw32$" | sed -E 's,]\s+.+$,,g' | sed 's,^\[,,g' | tr '\n' ',' | sed -E 's,(^.+$),export PLATNUMS=\1,' | sed 's/,$//' && echo)
$ echo $PLATNUMS
30,34,75,77
If you ever think you need grep+sed or 2 greps or 2 seds or any other combination then you should use 1 call to awk instead, and you never need grep or sed when you're using awk:
export PLATNUMS=$(XXX | awk -F'[][]' '/(i686-nptl-linux-gnu|i686-w64-mingw32|x86_64-unknown-linux-gnu|x86_64-w64-mingw32)$/{p=(p ? p "," : "") $2} END{print p}')
Btw in case it's useful, here's a couple of briefer regexps:
(i686-(nptl-linux-gnu|w64-mingw32)|x86_64-(unknown-linux-gnu|w64-mingw32))$
((i686-nptl|x86_64-unknown)-linux-gnu|(i686|x86_64)-w64-mingw32)$
and depending on your input data (since this will include combinations not provided by the above) you MIGHT only need:
(i686|x86_64)-(nptl|unknown|w64)-(linux-gnu|mingw32)$
Trying to grep a phrase out of multiple files as they are constantly populated (logs), but with hint as to which file was updated with the phrase.
For example:
grep bindaddr /vservers/*/var/log
gets me:
/vservers/11010/var/log:bindaddr=xxx.xxx.xxx.xxx
/vservers/12525/var/log:bindaddr=xxx.xxx.xxx.xxx
/vservers/12593/var/log:bindaddr=xxx.xxx.xxx.xxx
Which is cool, but I need this for tail -f.
tail -fn 100 /vservers/*/var/log | grep bindaddr
gets me the lines needed but no indicator in which file, so I need a mix of the two.
If you use -v in tail, you get a verbose mode: from man tab --> "always output headers giving file names". This way, whenever something happens in a file, you will get the header on the preceding line.
Together with this, you can use grep -B1 to show the match + the previous line.
All together, this should do:
tail -fvn 100 /vservers/*/var/log | grep -B1 bindaddr
Test
Doing this in one tab:
$ echo "hi" >> a2
$ echo "hi" >> a2
$ echo "hi" >> a1
$ echo "hi" >> a2
I got this in the other one:
$ tail -vfn 100 /tmp/a* | grep -B1 "h"
==> /tmp/a1 <==
==> /tmp/a2 <==
hi
hi
==> /tmp/a1 <==
hi
==> /tmp/a2 <==
hi
Something like this to put the filename in the front of each line from tail:
#!/bin/bash
# Arrange to kill all descendants on exit/interrupt
trap "kill 0" SIGINT SIGTERM EXIT
for f in *.txt; do
tail -f "$f" | sed "s/^/"$f": /" > /dev/tty &
done
# grep in stdin (i.e. /dev/tty)
grep bina -
I think some people are coming to this post looking for a way to display the filename while grepping the tail of multiple files:
for f in path/to/files*.txt; do echo $f; tail $f | grep 'SEARCH-THIS'; done;
This will display an output like this
filename1.txt
search result 1
search result 2
filenam2.txt
search result 3
search result 4
...