I'm trying to filter the output of git ls-files for some reporting commands, and then pipe that output to xargs. I was originally using -z on git ls-files which worked fine, but now that I'm piping to grep, I need to restore that functionality.
Trying the grep version -Z isn't doing anything to the actual output.
git ls-files | grep -Z -e={\*.js,\*.html}
The -Z option to grep only seems to apply to the filenames that contained the matched content not the content of the matching lines. That seems rather strange to me but that appears to be how it works.
You don't actually need grep here though (you didn't need the * in the original either it didn't add anything of value).
git ls-files takes a pattern to match against the filenames.
So you want to use this command here.
git ls-files -z -- '*.js' '*.html'
Related
I'm trying to do a grep in Microsoft Windows, using the MINGW64 shell v4.4.23(1). (That's what the title bar says. I assume this means MingW-W64.)
I want to list all files in a specified directory tree that have a certain filename extension and do not contain a certain string.
With the current directory set to the top of the tree I entered
grep -r -L thestring *.theextension
It lists only files in the current directory, not the tree.
I tried some variations and determined that grep is simply ignoring the -r option. It ignores --recursive, too.
But when I enter grep --help, it lists both -r and --recursive as valid options, with the expected meaning.
Is this a bug in the shell, or am I doing something stupid?
With grep -r -L thestring *.theextension you are telling grep to search recursively in any file or folder matching *.theextension. If you don't have any folders matching that you shouldn't expect it to go through any other folders. The -L flag doesn't mean it's going to look at anything not matching *.theextension, maybe that's what was confusing you...
while
grep -ir "xyz" * recursively searches through the directories and tell me that the text is present in ./x/y/z/abc.cpp
However ,
grep -ir "xyz" *.cpp offers no result.
Isn't the second command supposed to recursively grep all cpp files inside the directory ?
What am I missing here?
Grep will recurse through any directories you match with your glob pattern. (In your case, you probably do not have any directories that match the pattern "*.cpp") You could explicitly specify them: grep -ir "xyz" *.cpp */*.cpp */*/*.cpp */*/*/*.cpp, etc. You can also use the --include option (see the example below)
If you are using GNU grep, then you can use the following:
grep -ir --include "*.cpp" "xyz" .
The command above says to search recursively starting in current directory ignoring case on the pattern and to only search in files that match the glob pattern "*.cpp".
OR if you are on some other Unix platform, you can use this:
find ./ -type f -name "*.cpp" -print0 | xargs -0 grep -i "xyz"
If you are sure that none of your files have spaces in their names, you can omit the -print0 argument to find and the -0 to xargs
The command above says the following: find all files (-type f) under the current directory (./) that match the name glob/wildcard "*.cpp" (-name "*.cpp") and then print them out delimited by a null (-print0). That list of files found should be written to the stdin of the next command: xargs. xargs should read from stdin (default behavior) and split its input on nulls (-0) and then call the grep command with the specified options (grep -i "xyz") on that list of files.
If you are interested in learning more about why grep -ir "xyz" *.cpp does not work the way you think it should, you should search for "shell globbing" (here is a good first article on the subject). I'll also try to provide a quick explanation. When you type in the command grep -ir "xyz" *.cpp and hit enter, there are two programs that are involved in executing your command. The first program is your shell (and unless you've done something to customize things, you are probably usually the bash shell - if you've never heard of a shell or bash, that's where you should start looking, there are tons of good articles). Suffice it say that a shell is just a program that is designed to let you navigate the filesystem on your computer and run other programs. (In Windows, when you double click on an icon to launch a program, or open a folder to access a file, the program that you are running is explorer.exe and it is the Windows graphical shell). So, when you type the command grep -ir "xyz" *.cpp, before grep is run, the shell handles reading your command and does a few things. One of the things is does is expand glob patterns (things like *.txt or [0-9]+.pdf). Like I said, if you want to understand it, go read more about it, but the thing you should take away is that the grep command never sees the *.cpp. What happens is, the shell looks in the current directory for any files or directories with a name that match the pattern *.cpp and then replaces them on the command line BEFORE it runs the grep command. (If it doesn't find anything that matches, then it will leave the *.cpp there and grep will see it, but grep because doesn't normally do glob matching, this doesn't do anything for you).
Alternatively, when you type in grep -ir "xyz" *, what happens is that the shell replaces the * with the name of every file and directory in the current directory (because * matches anything). Let's say you had a directory that contained file1, file2, and dir1, and dir2, then the shell would perform its replacements and then execute a command that looked like this grep -ir "xyz" file1 file2 dir1 dir2, which means grep would search file1 and file2 for a line with the string xyz, and because of the -ir it also search recursively through dir1 and dir2 and search any files found for that string as well. Lastly, if you've followed everything I've said so far, then it will make sense to you that grep does have a way to use glob patterns on recursive searches, and that is to use the --include option, as in the command I described earlier: grep -ir --include "*.cpp" "xyz" ., and the reason why we put the *.cpp in quotes in that command is to prevent the shell from trying to expand the glob pattern before we run the command.
Is there a way to run shell commands without output buffering?
For example, hexdump file | ./my_script will only pass input from hexdump to my_script in buffered chunks, not line by line.
Actually I want to know a general solution how to make any command unbuffered?
Try stdbuf, included in GNU coreutils and thus virtually any Linux distro. This sets the buffer length for input, output and error to zero:
stdbuf -i0 -o0 -e0 command
The command unbuffer from the expect package disables the output buffering:
Ubuntu Manpage: unbuffer - unbuffer output
Example usage:
unbuffer hexdump file | ./my_script
AFAIK, you can't do it without ugly hacks. Writing to a pipe (or reading from it) automatically turns on full buffering and there is nothing you can do about it :-(. "Line buffering" (which is what you want) is only used when reading/writing a terminal. The ugly hacks exactly do this: They connect a program to a pseudo-terminal, so that the other tools in the pipe read/write from that terminal in line buffering mode. The whole problem is described here:
http://www.pixelbeat.org/programming/stdio_buffering/
The page has also some suggestions (the aforementioned "ugly hacks") what to do, i.e. using unbuffer or pulling some tricks with LD_PRELOAD.
You could also use the script command to make the output of hexdump line-buffered (hexdump will be run in a pseudo terminal which tricks hexdump into thinking its writing its stdout to a terminal, and not to a pipe).
# cf. http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe/
stty -echo -onlcr
script -q /dev/null hexdump file | ./my_script # FreeBSD, Mac OS X
script -q -c "hexdump file" /dev/null | ./my_script # Linux
stty echo onlcr
One should use grep or egrep "--line-buffered" options to solve this. no other tools needed.
I am trying to create a script, and one part requires showing lines with numeric values.
My basic syntax is:
echo $i | grep [0-9]
For example, I set i=12345, it should output 12345.
But on one server, it doesn't output anything (exactly the same commands).
I do not know how to Google this issue, I have tried "grep output different on other server", to no avail.
When using a regexp, either use egrep or grep -e to make sure the pattern is not treated as a plain string.
maybe it's a shell issue? some shells interpert [] differently
try
echo "1234" | grep "[0-9]"
(with quotes)
also try
grep --version
to see if there is a different grep version
I have a Git repository (originally CVS, then SVN, now Git) containing a Rails project that has been deployed on Linux for a while now. Everything seems to run fine.
Now that I've converted to git, I see that many of my files in the repository contain CRLF line endings. I'd love for it to all be consistent (LF), but not at the expense of loosing the edit history of every file that has CRLF line endings.
Can you think of any reason I can't leave the files as they are? I seem to remember there being a problem with shell scripts or cron files or something that didn't respond to CRLF very well.
Also, I know all about the Git options core.autocrlf and core.safecrlf, But is there some way to have it convert all text files from CRLF to LF on checkout (for the linux side) ... i.e. a core.autolf option or something similar?
If it is ok for you to rewrite your repository's history (see problems with rewriting history) you could use git filter-branch to convert CRLF to LF:
git filter-branch --tree-filter 'find . -path './.git' -prune -o -type f -exec dos2unix \{} \;' HEAD
Note that if you have binary files in your repository you will have to refine the find command to exclude them.
A comment to davitenio's answer and Daniel Beardsley's comment; I believe you could use this little program as a wrapper around dos2unix:
#!/bin/sh
for f in $#; do
if [ $(file -b -n -i -m /dev/null $f | grep -c "text") -gt 0 ]; then
dos2unix $f
fi
done
although there is probably still some corner case that will fail.