I am trying to get strings that are separated by colons from a file with grep. I have managed fine so far but I ran into a problem where grep is just ignoring one of the characters I am trying to include in the search.
The file I am searching contains this line
configname:user:ip:password:macaddr
and the command I am running is grep -o "configname:.*:" sshutil_config
I thought this would find "configname:user:" but all it does is remove "macaddr" from the output. configname:user:ip:password:
The username will change and so I never know what it is so I can't grep specifically for it but I will know the configname and so I am trying to search for the username using it. I need to get the username out of the file as input and save it to a variable using usr=$(grep -o "whatever_this_needs_to_be" sshutil_config)
Thanks in advance,
-A\\/
Change "configname:.*:" to "configname:[^:]*:"
This is also possible if you don't want to use grep
user="$( cut -d ':' -f 2 <<< "$data" )"
I have a file, for example, "queries.txt" that has hard return separated strings. I want to use this list to find matches in a second file, "biglist.txt".
"biglist.txt" may have multiple matches for each string in "queries.txt". I want to return only the first hit for each query and write this to another file.
grep -m 1 -wf queries.txt biglist.txt > output
only gives me one line in output. I should have output that is the same number of lines as queries.txt.
Any suggestions for this? Many thanks! I searched for past questions but did not find one that was exactly the same sort of case after a few minutes of reading.
If you want to "reset the counter" after each file, you could do
cat queries.txt | xargs -I{} grep -m 1 -w {} biglist.txt > output
This uses xargs to call grep once for each line in the input… should do the trick for you.
Explanation:
cat queries.txt - produce one "search word" per line
xargs -I{} - take the input one line at a time, and insert it at {}
grep -m 1 -w - find only one match of a whole word
{} - this is where xargs inserts the search term (once per call)
biglist.txt - the file to be searched
> output - the file where the result is to be written
An alternate method without xargs (which one should indeed learn):
(this method assumes there are no spaces in the lines in queries.txt)
cat queries.txt | while read target; do grep -m 1 $target biglist.txt; done > outr
I might not fully understand your question, but it sounds like something like this might work.
cat queries.txt | while read word; do grep "$word" biglist.txt | tee -a output.txt; done
so thanks to this forum, I currently have this code, which takes an output from a programme I have and saves it in a file:
#!usr/bin/python
import os
os.chdir('./P574/J0998-1034')
os.system('vap -c freq *.SFTC > 1400list.txt')
I wanted to add a filter (so take only lines that contained "1369.000", so I amended the last line to:
os.system('vap -c freq *.SFTC | egrep 1369.000 > 1400listfilt.txt')
But I really want it to take lines that contain EITHER "1369.000" OR "1433.000". I tried:
os.system('vap -c freq *.SFTC | egrep 1369.000|1433.000 > 1400listfilt.txt' )
But I got the error message: "sh: 1433.000: command not found
egrep: write error: Broken pipe"
How can I make it check for two values? Also.. is this the best way to do what I am trying to do?
Thank you!
I would surround the arguments in single quotes as such:
egrep '(1369.000|1433.000)'
The shell is telling you that it could not redirect the output of egrep to the program 1433.000 which doesn't exist.
I have the following file:
asdasd
asd
asd
incompatible:svcnotallowed:svc\:/network/bmb/clerver\:default
incompatible:svcnotallowed:svc\:/network1/bmb/clerver\:default
incompatible:svcnotallowed:svc\:/network2/bmb/clerver\:default
asdasd
asd
asd
as
And now suppose I have the two variables v1="incompatible:svcnotallowed:" and v2="svc\:/network1/bmb/clerver\:default".
I would like to search the entire file using v1 and v2. I know this is a problem caused due to the file having a'\' in it. I just dont know how to eliminate it. I have tried storing v1 and v2 (both variable contents and grep usage) using single quotes, but in vain.
This is the series of commands I have tried :
grep "$v1$v2" file
grep '$v1$v2' file
I need this to work in KSH
please let me know the right way to use grep in this scenario.
Thanks.
grep -F "$v1$v2" file should do the trick -- with the -F option, it treats the pattern as a fixed string, so backslashes don't get interpreted as escapes or backreferences.
But fgrep "$v1$v2" file would probably be the most portable solution. As tomkaith13 notes in his comment, the -F option to grep isn't universally supported. On Solaris, the default grep doesn't support -F, but the version in /usr/xpg4/bin does.
Since you are using ksh, you can just use it to read the files
v1="incompatible:svcnotallowed:"
v2="svc\:/network1/bmb/clerver\:default"
while read -r line
do
case "$line" in
"$v1$v2" ) echo "$line";;
esac
done < file
Is there a way to make grep output "words" from files that match the search expression?
If I want to find all the instances of, say, "th" in a number of files, I can do:
grep "th" *
but the output will be something like (bold is by me);
some-text-file : the cat sat on the mat
some-other-text-file : the quick brown fox
yet-another-text-file : i hope this explains it thoroughly
What I want it to output, using the same search, is:
the
the
the
this
thoroughly
Is this possible using grep? Or using another combination of tools?
Try grep -o:
grep -oh "\w*th\w*" *
Edit: matching from Phil's comment.
From the docs:
-h, --no-filename
Suppress the prefixing of file names on output. This is the default
when there is only one file (or only standard input) to search.
-o, --only-matching
Print only the matched (non-empty) parts of a matching line,
with each such part on a separate output line.
Cross distribution safe answer (including windows minGW?)
grep -h "[[:alpha:]]*th[[:alpha:]]*" 'filename' | tr ' ' '\n' | grep -h "[[:alpha:]]*th[[:alpha:]]*"
If you're using older versions of grep (like 2.4.2) which do not include the -o option, then use the above. Else use the simpler to maintain version below.
Linux cross distribution safe answer
grep -oh "[[:alpha:]]*th[[:alpha:]]*" 'filename'
To summarize: -oh outputs the regular expression matches to the file content (and not its filename), just like how you would expect a regular expression to work in vim/etc... What word or regular expression you would be searching for then, is up to you! As long as you remain with POSIX and not perl syntax (refer below)
More from the manual for grep
-o Print each match, but only the match, not the entire line.
-h Never print filename headers (i.e. filenames) with output lines.
-w The expression is searched for as a word (as if surrounded by
`[[:<:]]' and `[[:>:]]';
The reason why the original answer does not work for everyone
The usage of \w varies from platform to platform, as it's an extended "perl" syntax. As such, those grep installations that are limited to work with POSIX character classes use [[:alpha:]] and not its perl equivalent of \w. See the Wikipedia page on regular expression for more
Ultimately, the POSIX answer above will be a lot more reliable regardless of platform (being the original) for grep
As for support of grep without -o option, the first grep outputs the relevant lines, the tr splits the spaces to new lines, the final grep filters only for the respective lines.
(PS: I know most platforms by now would have been patched for \w.... but there are always those that lag behind)
Credit for the "-o" workaround from #AdamRosenfield answer
It's more simple than you think. Try this:
egrep -wo 'th.[a-z]*' filename.txt #### (Case Sensitive)
egrep -iwo 'th.[a-z]*' filename.txt ### (Case Insensitive)
Where,
egrep: Grep will work with extended regular expression.
w : Matches only word/words instead of substring.
o : Display only matched pattern instead of whole line.
i : If u want to ignore case sensitivity.
You could translate spaces to newlines and then grep, e.g.:
cat * | tr ' ' '\n' | grep th
Just awk, no need combination of tools.
# awk '{for(i=1;i<=NF;i++){if($i~/^th/){print $i}}}' file
the
the
the
this
thoroughly
grep command for only matching and perl
grep -o -P 'th.*? ' filename
I was unsatisfied with awk's hard to remember syntax but I liked the idea of using one utility to do this.
It seems like ack (or ack-grep if you use Ubuntu) can do this easily:
# ack-grep -ho "\bth.*?\b" *
the
the
the
this
thoroughly
If you omit the -h flag you get:
# ack-grep -o "\bth.*?\b" *
some-other-text-file
1:the
some-text-file
1:the
the
yet-another-text-file
1:this
thoroughly
As a bonus, you can use the --output flag to do this for more complex searches with just about the easiest syntax I've found:
# echo "bug: 1, id: 5, time: 12/27/2010" > test-file
# ack-grep -ho "bug: (\d*), id: (\d*), time: (.*)" --output '$1, $2, $3' test-file
1, 5, 12/27/2010
cat *-text-file | grep -Eio "th[a-z]+"
You can also try pcregrep. There is also a -w option in grep, but in some cases it doesn't work as expected.
From Wikipedia:
cat fruitlist.txt
apple
apples
pineapple
apple-
apple-fruit
fruit-apple
grep -w apple fruitlist.txt
apple
apple-
apple-fruit
fruit-apple
I had a similar problem, looking for grep/pattern regex and the "matched pattern found" as output.
At the end I used egrep (same regex on grep -e or -G didn't give me the same result of egrep) with the option -o
so, I think that could be something similar to (I'm NOT a regex Master) :
egrep -o "the*|this{1}|thoroughly{1}" filename
To search all the words with start with "icon-" the following command works perfect. I am using Ack here which is similar to grep but with better options and nice formatting.
ack -oh --type=html "\w*icon-\w*" | sort | uniq
You could pipe your grep output into Perl like this:
grep "th" * | perl -n -e'while(/(\w*th\w*)/g) {print "$1\n"}'
grep --color -o -E "Begin.{0,}?End" file.txt
? - Match as few as possible until the End
Tested on macos terminal
$ grep -w
Excerpt from grep man page:
-w: Select only those lines containing matches that form whole words. The test is that the matching substring must either be at the beginning of the line, or preceded by a non-word constituent character.
ripgrep
Here are the example using ripgrep:
rg -o "(\w+)?th(\w+)?"
It'll match all words matching th.