I wanted to search through the output of ssh-keygen by piping it through grep. To my surprise, that didn't work, so I tried directing the contents to a text file like so: ssh-keygen --help > temp.txt and that didn't work either. I've concluded that ssh-keygen --help does not print to stdout. So I have a specific question and general question please:
Specific: How do I get ssh-keygen --help to end up in stdout?
General: What is going on here? If not to stdout, then to what/where are the contents being 'directed', and how would I have gone about figuring that out?
Related
I am a beginner of bash script. I just started to write a script where it checks the contents of b.txt can all be found in a.txt. (line by line preferably). My code is as following:
grep -Ffw b.txt a.txt
As you can see, I want to do fixed string instead of REGEX, I want to check everything from the b.txt file, because there are some strings inside the b.txt and I want to check if all of them exist in a.txt. And I also want to match the whole word only of course. So these are the requirements, however when I run this command it returns me an error says: grep: w: No such file or directory
I am thinking that maybe there are some limitations of the flags in bash? Sorry I am not really familiar with the language, didn't read much about the MAN page etc. If anyone could help me to solve the puzzle it would be appreciated :) In addition, i think if possible I would like to add a -q to surpress the output when there is a match also, right now I didn't add it in the example since it couldn't make it through with 3 flags even. So can anyone give me some hints here? Thanks in advance!
Hereby some explanation from the manpage:
OPTIONS
Generic Program Information
...
-F, --fixed-strings
Interpret PATTERNS as fixed strings, ...
-f FILE, --file=FILE
Obtain patterns from FILE, ...
-w, --word-regexp
Select only those lines ...
As you can see, the options -F and -w are indicated ending immediately (hence the comma in -F, and -w,), but the -f switch is followed by FILE, with means they belong together.
I you want to preserve the order Ffw, that's possible, but then you need to do something like:
grep -Ff b.txt -w a.txt
As mentioned by #kvantour, the solution is simply placing the -f before the b.txt file. grep -Fwf b.txt a.txt
Should have thought it when it says 'no such file or directory' as it was a clear indication that flags after the -f were treated as the path already.
Dipping my toes into Bash coding for the first time (not the most experienced person with Linux either) and I'm trying to read the version from the version.php inside a container at:
/config/www/nextcloud/version.php
To do so, I run:
docker exec -it 1c8c05daba19 grep -eo "(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?" /config/www/nextcloud/version.php
This uses a semantic versioning RegEx pattern (I know, a bit overkill, but it works for now) to read and extract the version from the line:
$OC_VersionString = '20.0.1';
However, when I run the command it tells me No such file or directory, (I've confirmed it does exist at that path inside the container) and then proceeds to spit out the entire contents of the file it just said doesn't exist?
grep: (0|[1-9]\d*).(0|[1-9]\d*).(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-])(?:.(?:0|[1-9]\d|\d*[a-zA-Z-][0-9a-zA-Z-]))))?(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?: No such file or directory
/config/www/nextcloud/version.php:$OC_Version = array(20,0,1,1);
/config/www/nextcloud/version.php:$OC_VersionString = '20.0.1';
/config/www/nextcloud/version.php:$OC_Edition = '';
/config/www/nextcloud/version.php:$OC_VersionCanBeUpgradedFrom = array (
/config/www/nextcloud/version.php: 'nextcloud' =>
/config/www/nextcloud/version.php: 'owncloud' =>
/config/www/nextcloud/version.php:$vendor = 'nextcloud';
Anyone able to spot the problem?
Update 1:
For the sake of clarity, I'm trying to run this from a bash script. I just want to fetch the version number from that file, to use it in other areas of the script.
Update 2:
Responding to the comments, I tried to login to the container first, and then run the grep, and still get the same result. Then I cat that file and it shows it's contents no problem.
Many containers don't have the GNU versions of Unix tools and their various extensions. It's popular to base containers on Alpine Linux, which in turn uses a very lightweight single-binary tool called BusyBox to provide the base tools. Those tend to have the set of options required in the POSIX specs, and no more.
POSIX grep(1) in particular doesn't have an -o option. So the command you're running is
grep \
-eo \ # specify "o" as the regexp to match
"(regexps are write-only)" \ # a filename
/config/www/nextcloud/version.php # a second filename
Notice that the grep output in the interactive shell only contains lines with the letter "o", but not for example the line just containing array.
POSIX grep doesn't have an equivalent for GNU grep's -o option
Print only the matched (non-empty) parts of matching lines, with each such part on a separate output line. Output lines use the same delimiters as input....
but it's easy to do that with sed(1) instead. Ask it to match some stuff, the regexp in question, and some stuff, and replace it with the matched group.
sed -e 's/.*\(any regexp here\).*/\1/' input-file
(POSIX sed only accepts basic regular expressions, so you'll have to escape more of the parentheses.)
Well, for any potential future readers, I had no luck getting grep to do it, I'm sure it was my fault somehow and not grep's, but thanks to the help in this post I was able to use awk instead of grep, like so:
docker exec -it 1c8c05daba19 awk '/^\$OC_VersionString/ && match($0,/\047[0-9]+\.[0-9]+\.[0-9]+\047/){print substr($0,RSTART+1,RLENGTH-2)}' /config/www/nextcloud/version.php
That ended up doing exactly what I needed:
It logs into a docker container.
Scans and returns just the version number from the line I am looking for at: /config/www/nextcloud/version.php inside the container.
Exits stage left from the container with just the info I needed.
I can get right back to eating my Hot Cheetos.
I'm writing a script, and I need to look up a command on the user's $PATH and get the full path to the command. The problem is that I don't know what the user's login shell is, or what strange stuff might be in their do files. I'm using the bourne shell for my simple little script because it needs to run on some older Solaris platforms that might not have bash.
Some implementations of "which" and "whence" will source the user's dot files, and that isn't really portable to all users. I'd love a simple UNIX utility that would just do the basic job of scanning PATH for an executable and reporting the full path of the first match.
But I'll settle for any /bin/sh solution that is stable for all users.
I'm looking for a solution that is better than writing my own /bin/sh loop that chops up $PATH and searches it one line at a time. It would seem that this is common enough that there should be an reusable way to do it.
My first approximation of the "long way" is this:
IFS=:
for i in $PATH; do
if [ -x $i/$cmd ]; then
echo $i/$cmd
fi
done
Is there something simpler and portable?
The answer seems to be the 'type' built-in.
% /bin/sh
$ type ls
ls is /bin/ls
Maybe the whereis command will work for you?
whereis -b -B `echo $PATH | sed 's/:/ /g'` -f [commands]
e.g. on my computer, this works:
whereis -b -B `echo $PATH | sed 's/:/ /g'` -f find man fsc
And results in:
find: /usr/bin/find
man: /usr/bin/man
fsc: /opt/FSharp-2.0.0.0/bin/fsc.exe /opt/FSharp-2.0.0.0/bin/fsc
One caveat from the whereis man page:
Since whereis uses chdir(2V) to run faster, pathnames given
with the -M, -S, or -B must be full; that is, they must begin
with a `/'.
This question is answered in details here: https://unix.stackexchange.com/questions/85249/why-not-use-which-what-to-use-then. Bottom line: use command -v ls.
Is there a way to run shell commands without output buffering?
For example, hexdump file | ./my_script will only pass input from hexdump to my_script in buffered chunks, not line by line.
Actually I want to know a general solution how to make any command unbuffered?
Try stdbuf, included in GNU coreutils and thus virtually any Linux distro. This sets the buffer length for input, output and error to zero:
stdbuf -i0 -o0 -e0 command
The command unbuffer from the expect package disables the output buffering:
Ubuntu Manpage: unbuffer - unbuffer output
Example usage:
unbuffer hexdump file | ./my_script
AFAIK, you can't do it without ugly hacks. Writing to a pipe (or reading from it) automatically turns on full buffering and there is nothing you can do about it :-(. "Line buffering" (which is what you want) is only used when reading/writing a terminal. The ugly hacks exactly do this: They connect a program to a pseudo-terminal, so that the other tools in the pipe read/write from that terminal in line buffering mode. The whole problem is described here:
http://www.pixelbeat.org/programming/stdio_buffering/
The page has also some suggestions (the aforementioned "ugly hacks") what to do, i.e. using unbuffer or pulling some tricks with LD_PRELOAD.
You could also use the script command to make the output of hexdump line-buffered (hexdump will be run in a pseudo terminal which tricks hexdump into thinking its writing its stdout to a terminal, and not to a pipe).
# cf. http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe/
stty -echo -onlcr
script -q /dev/null hexdump file | ./my_script # FreeBSD, Mac OS X
script -q -c "hexdump file" /dev/null | ./my_script # Linux
stty echo onlcr
One should use grep or egrep "--line-buffered" options to solve this. no other tools needed.
I am a newbie to shell scripting and to Linux environment as well.
In my project I am trying to search for following text from the httpd.conf file
<Directory '/somedir/someinnerdir'>
AllowOverride All
</Directory>
and then remove this text and again rewrite the same text.
The reason to do this rewriting is that the script will be run on first installation of the web app, but it may again be run some time later as other part of this shell script is performing other tasks as well. So for first time this text wont be found and will simply be written but later again when script is run this text will be found and will need to be removed and the written again.
So the part of my script with which I am trying to achieve this is something like :
grep -ve "<Directory '/somedir/someinnerdir'>\\nAllowOverride All\\n</Directory>" /etc/httpd/conf/httpd.conf > tmp_direct
echo -e "<Directory '/somedir/someinnerdir'>\\nAllowOverride All\\n</Directory>" >> tmp_direct
mv tmp_direct /etc/httpd/conf/httpd.conf
I dont have the code in front of me currently so there may be some syntactical errors above but the logic/coding is same.
Above code fragment is not able to do what I want to achieve as the grep command doesnt support multiline searching.
My OS is Fedora 8.
Can you please suggest something in this code to achieve what is needed or may be some other alternative.
Any help in this regard will be highly appreciated.
Thanks in advance.
Thanks for your replies.
Sorry for the previous bad code. Its corrected now.
Charlie and i-moan, due to workability constraints I wont be able to implement sed or perl as it will need to be added to the environment we will distribute this project in.
Steve, I want to do the check for multiple lines. I didnt put it in code blocks so it removed the directory tags. :(
So I will need to find some other way out.
Thanks again.
Best regards.
This sounds like a job for sed(1). What you want will be very close to
sed -i.bak -e '/AllowOverride All/i# ' file
to comment out the line (remember httpd.conf has comments) followed by
sed -i.bak -e 's/# AllowOverride All/AllowOverride All ' file
to pujt it back.
WARNING I haven't tried it, you want to read the man page and test it youself.
If you want to make sure you're matching an entire line, you could do:
grep -v "^AllowOverride All$"
The ^ matches the start of line and $ matches the end of line.
Does that help? I'm not exactly clear on what you're trying to do with the grep.
I'm unsure I understand the question.
Correct me if I'm wrong but your trying to add 'AllowOverride All' to the httpd.conf on the 1st running of your script. If so do you want it enabled?
Then on a 2nd pass you may want to enable / disable the command?
If I remember correctly 'AllowOverride' is only allowed in a < Directory > section So it only makes sense to add / enable in < Directory >< /Directory > definition:
If above is true you could do this in perl..
1st run: to add
perl -0777 -pi
-e 's{(.?(?!AllowOverride\s+All).?)}[\nAllowOverride
All\n$1]ixgs' httpd.conf
This will add 'AllowOverride All' to a Directory section only if it does not already contain one.
2nd run: to alter
On the 2nd run you could comment out as Charlie's suggested.. or if you like perl
perl -0777 -pi
-e's{(.?)(AllowOverride\s+All)(.?)}[$1#$2$3]ixgs' httpd.conf
and to comment in
perl -0777 -pi
-e's{(.?)(#AllowOverride\s+All)(.?)}[$1AllowOverride\s+All$3]ixgs'
httpd.conf
The regex s{}[] is quite verbose and there are better regex's to do the same thing but this is easier ( hopefully ) for a beginner to understand :).
Explanation of the perl command options
-0777 force Perl to read the whole file in one shot because 0777 is not a legal character value
-p places a loop around your script
-i lets you edit files in-place ( so your command in '-e' change the file )
-e lets you specify a single line of code on the command line
In that single line we have a regex. see perldoc perlre for info on that.
Hope that helps