In bash, php/{composer,sismo} expands to php/composer php/sismo. Is there any way to do this with /bin/sh (which I believe is dash), the system shell ? I'm writing git hooks and would like to stay away from bash as long as I can.
You can use printf.
% printf 'str1%s\t' 'str2' 'str3' 'str4'
str1str2 str1str3 str1str4
There doesn't seem to be a way. You will have to use loops to generate these names, perhaps in a function. Or use variables to substitute common parts, maybe with "set -u" to prevent typos.
I see that you prefer dash for performance reasons, however you don't seem to provide any numbers to substantiate your decision. I'd suggest you measure actual performance difference and reevaluate. You might be falling for premature optimization, as well. Consider how much implementation and debugging time you'll save by using Bash vs. possible performance drop.
I really like the printf solution provided by #mikeserv, but I thought I'd provide an example using a loop.
The below would probably be most useful if you wish to execute one command for each expanded string, rather than provide both strings as args to the same command.
for X in composer sismo; do
echo "php/$X" # replace 'echo' with your command
done
You could, however, rewrite it as
ARGS="$(for X in composer sismo; do echo "php/$X"; done)"
echo $ARGS # replace 'echo' with your command
Note that $ARGS is unquoted in the above command, and be aware that this means that its content is wordsplitted (i.e. if any your original strings contain spaces, it will break).
Related
I’m trying to improve a snippet for Gedit that helps me write shell scripts.
Currently, the snippet encloses the name of a variable into double quotes surrounding curly brackets preceded with a dollar sign. But to make the letters uppercase, I have to switch to the caps-lock mode or hold down a shift key when entering the words. Here is the code of the snippet:
"\${$1}"
I would like that the snippet makes the letters uppercase for me. To do that, I need to know how to make text uppercase and change the content of a placeholder.
I have carefully read the following articles:
https://wiki.gnome.org/Apps/Gedit/Plugins/Snippets
https://blogs.gnome.org/jessevdk/2009/12/06/about-snippets/
https://www.marxists.org/admin/volunteers/gedit-sed.htm
How do you create a date snippet in gedit?
But I still have no idea how to achieve what I want — to make the letters uppercase. I tried to use the output of shell programs, a Python script, the regular expressions — the initial text in the placeholder is not changed. The last attempt was the following (for clarity, I removed the surrounding double-quotes and the curly brackets with the dollar — working just on the letter case):
${1}$<[1]: return $1.upper()>
But instead of MY_VARIABLE I get my_variableMY_VARIABLE.
Perhaps, the solution is obvious, but I cannot get it.
I did it! The solution found!
Before all, I have to say that I don’t count the solution as correct or corresponding to the ideas of the Gedit editor. It’s a dirty hack, of course. But, strangely, there is no way to change the initial content of placeholders in the snippets — haven’t I just found a standard way to do that?
So. If they don’t allow us to change the text in placeholders, let’s ask the system to do that.
The first thought that stroke me was to print backspace characters. There are two ways to do that: a shell script and a python script. The first approach might look like: $1$(printf '\b') The second one should do the same: $1$<[1]: return '\b'> But both of them don’t work — Gedit prints surrogate squares instead of real backspace characters.
Thus, xdotool is our friend! So is metaprogramming! You will not believe — metaprogramming in a shell script inside a snippet — sed will be writing the scenario for xdotool. Also, I’ve added a feature that changes spaces to underscores for easier typing. Here is the final code of the snippet:
$1$(
eval \
xdotool key \
--delay 5 \
`echo "${1}" | sed "s/./ BackSpace/g;"`
echo "\"\${${1}}\"" \
| tr '[a-z ]' '[A-Z_]'
)$0
Here are some explanations.
Usually, I never use backticks in my scripts because of some troubles and incompatibilities. But now is not the case! It seems Gedit cannot interpret the $(...) constructions correctly when they are nested, so I use the backticks here.
A couple of words about using the xdotool command. The most critical part is the --delay option. By default, it’s 12 milliseconds. If I leave it as is, there will be an error when the length of the text in the placeholder is quite long. Not to mention the snippet processing becomes slow. But if I set the time interval too small, some of the emulated keystrokes sometimes will be swallowed somewhere. So, five milliseconds is the delay that turns out optimal for my system.
At last, as I use backspaces to erase the typed text, I cannot use template parts outside the placeholder. Thus, such transformations must be inside the script. The complex heap after the echo command is what the template parts are.
What the last tr command does is the motivator of all this activity.
It turns out, Gedit snippets may be a power tool. Good luck!
I am trying to understand and read the man page. Yet everyday I find more inconsistent syntax and I would like some clarification to whether I am misunderstanding something.
Within the man page, it specifies the syntax for grep is grep [OPTIONS] [-e PATTERN]... [-f FILE]... [FILE...]
I got a working example that recursively searches all files within a directory for a keyword.
grep -rnw . -e 'memes
Now this example works, but I find it very inconsistent with the man page. The directory (Which the man page has written as [FILE...] but specifies the use case for if file == directory in the man page) is located last. Yet in this example it is located after [OPTIONS] and before [-e PATTERN].... Why is this allowed, it does not follow the specified regex fule of using this command?
Why is this allowed, it does not follow the specified regex fule of using this command?
The lines in the SYNOPSIS section of a manpage are not to be understood as strict regular expressions, but as a brief description of the syntax of a utility's arguments.
Depending on the particular application, the parser might be more or less flexible on how it accepts its options. After all, each program can implement whatever grammar they like for their arguments. Therefore, some might allow options at the beginning, at the end, or even in-between files (typically with ways to handle ambiguity that may arisa, e.g. reading from the standard input with -, filenames starting with -...).
Now, of course, there are some ways to do it that are common. For instance, POSIX.1-2017 12.1 Utility Argument Syntax says:
This section describes the argument syntax of the standard utilities and introduces terminology used throughout POSIX.1-2017 for describing the arguments processed by the utilities.
In your particular case, your implementation of grep (probably GNU's grep) allows to pass options in-between the file list, as you have discovered.
For more information, see:
https://unix.stackexchange.com/questions/17833/understand-synopsis-in-manpage
Are there standards for Linux command line switches and arguments?
https://www.gnu.org/software/libc/manual/html_node/Getopt-Long-Options.html
You can also leverage .
grep ‘string’ * -lR
In a Windows batch file, when you do the following:
set myvar="c:\my music & videos"
the variable myvar is stored with the quotes included. Honestly I find that very stupid. The quotes are just to tell where the string begins and ends, not to be stored as part of the value itself.
How can I prevent this from happening?
Thanks.
set "myvar=c:\my music & videos"
Notice the quotes start before myvar. It's actually that simple.
Side note: myvar can't be echoed afterwards unless it's wrapped in quotes because & will be read as a command separator, but it'll still work as a path.
http://ss64.com/nt/set.html under "Variable names can include Spaces"
This is the correct way to do it:
set "myvar=c:\my music & videos"
The quotes will not be included in the variable value.
It depends on how you want to use the variable. If you just want to use the value of the variable without the quotes you can use either delayed expansion and string substitution, or the for command:
#echo OFF
SETLOCAL enabledelayedexpansion
set myvar="C:\my music & videos"
As andynormancx states, the quotes are needed since the string contains the &. Or you can escape it with the ^, but I think the quotes are a little cleaner.
If you use delayed expansion with string substitution, you get the value of the variable without the quotes:
#echo !myvar:"=!
>>> C:\my music & videos
You can also use the for command:
for /f "tokens=* delims=" %%P in (%myvar%) do (
#echo %%P
)
>>> C:\my music & videos
However, if you want to use the variable in a command, you must use the quoted value or enclose the value of the variable in quotes:
Using string substitution and delayed expansion to use value of the variable without quotes, but use the variable in a command:
#echo OFF
SETLOCAL enabledelayedexpansion
set myvar="C:\my music & videos"
md %myvar%
#echo !myvar:"=! created.
Using the for command to use the value of the variable without quotes, but you'll have to surround the variable with quotes when using it in commands:
#echo OFF
set myvar="C:\my music & videos"
for /f "tokens=* delims=" %%P in (%myvar%) do (
md "%%P"
#echo %%P created.
)
Long story short, there's really no clean way to use a path or filename that contains embedded spaces and/or &s in a batch file.
Use jscript.
Many moons ago (i.e. about 8 years give or take) I was working on a large C++/VB6 project, and I had various bits of Batch Script to do parts of the build.
Then someone pointed me at the Joel Test, I was particularly enamoured of point 2, and set about bringing all my little build scripts into one single build script . . .
and it nearly broke my heart, getting all those little scripts working together, on different machines, with slightly different setups, ye Gods it was dreadful - particularly setting variables and parameter passing. It was really brittle, the slightest thing would break it and require 30 minutes of tweaking to get going again.
Eventually - I can be stubborn me - I chucked the whole lot in and in about a day re-wrote it all in JavaScript, running it from the command prompt with CScript.
I haven't looked back. Although these days it's MSBuild and Cruise Control, if I need to do something even slightly involved with a batch script, I use jscript.
The Windows command interpreter allows you to use the quotes around the entire set command (valid in every version of windows NT from NT 4.0 to Windows 2012 R2)
Your script should just be written as follows:
#echo OFF
set "myvar=C:\my music & videos"
Then you may put quotes around the variables as needed.
Working with the CMD prompt can seem esoteric at times, but the command interpreter actually behaves pretty solidly in obeying it's internal logic, you just need to re-think things.
In fact, the set command does not require you to use quotes at all, but both the way you are doing your variable assignment and the way the ,method of using no quotes can cause you to have extra spaces around your variable which are hard to notice when debugging your script.
e.g. Both of the below are technically Valid, but you can have trailing spaces, so it's not a good practice:
set myvar=some text
set myvar="some text"
e.g. Both of the below are good methods for setting variables in Windows Command interpreter, however the double quote method is superior:
set "myvar=Some text"
(set myvar=Some value)
Both of these leave nothing to interpretation the variable will have exactly the data you are looking for.
strong text However, for your purposes, only the quoted method will work validly because you are using a reserved character
Thus, you would use:
set "myvar=c:\my music & videos"
However, even though the variable IS correctly set to this string, when you ECHO the sting the command interpreter will interpret the ampersand as the keyword to indicate another statement follows.
SO if you want to echo the string from the variable the CMD interpreter still needs to be told it's a text string, or if you do not want the quotes to show you have to do one of the following:
echo the variable WITH Quotes:
Echo."%myvar%"
echo the variable WITHOUT Quotes:
Echo.%myvar:&=^&%
<nul SET /P="%myvar%"
In the above two scenarios you can echo the string with no quotes just fine. Example output below:
C:\Admin> Echo.%myvar:&=^&%
C:\my music & videos
C:\Admin> <nul SET /P="%myvar%"
C:\my music & videos
C:\Admin>
Try using the escape character '^', e.g.
set myvar=c:\my music ^& videos
You'll have you be careful when you expand myvar because the shell might not treat the & as a literal. If the above doesn't work, try inserting a caret into the string too:
set myvar=c:\my music ^^^& videos
Two solutions:
Don't use spaces or other characters that are special to the command interpreter in path names (directory or file names). If you use only letters, numbers, underscores, and hyphens (and a period before the extension to identify the file type), your scripting life will become immeasurably simpler.
I have written and otherwise collected a plethora of tools over the years, including a DOS utility that will rename files. (It began as something that just removed spaces from filenames, but morphed into something that will replace characters or strings within the filenames, even recursively.) If anyone's interested, I will get my long-neglected web site up and running and post this and others.
That said, variables aren't just for holding pathnames, so...
As others have already pointed out, SET "myvar=c:\my music & videos" is the correct workaround for such a variable value. (Yes, I said workaround. I agree that your initial inclination to just quote the value ("my music & videos") is far more intuitive, but it is what it is, as they say.
I have a log file that may be very large (10+ GB). I'd like to find the last occurrence of an expression. Is it possible to do this with standard posix commands?
Here are some potential answers, from similar questions, that aren't quite suitable.
Use tail -n <x> <file> | grep -m 1 <expression>: I don't know how far back the expression is, so I don't know what <x> would be. It could be several GB previous, so then you'd be tailing the entire file. I suppose you could loop and increment <x> until it's found, but then you'd be repeatedly reading the last part of the file.
Use tac <file> | grep -m 1 <expression>: tac reads the entire source file. It might be possible to chain something on to sigkill tac as soon as some output is found? Would that be efficient?
Use awk/sed: I'm fairly sure these both always start from the top of the file (although I may be wrong, my sed-fu is not strong).
"There'd be no speed up so why bother": I think that's incorrect, since file systems can seek to the end of a file without reading the whole thing. There'd be a little trial and error/buffering to find each new line, but that shouldn't slow things down much, compared to reading (e.g.) 10 GB that are never used.
Write a python/perl script to do it: this is my fall-back if no one can suggest anything better. I'd rather stick to something that can be done straight through the command line, since I'm executing it straight through ssh, and I'd rather not have to upload a script file as well. Using mmap's rfind() in python, I think we can do it in a few lines, provided the expression to find is static (which mine, unfortunately, is not). A regex requires a bit more work, something like this.
If it helps, the expression is anchored at the start of a line, eg: "^foo \d+$".
Whatever script you write will almost certainly be slower than:
tac file | grep -m 1 '^foo [0-9][0-9]*$'
This awk script will search through the whole file and print the last line matching the given /pattern/:
$ awk '/pattern/ { line=$0 } END { print $line }' gigantic.log
Using tac will be a better option (this uses GNU sed to output the first (i.e. last) found match with '/pattern/', after which it terminates, killing the pipeline):
$ tac gigantic.log | gsed -n '/pattern/{p;q}'
Using Perl or C or some other language, you could seek to the end of the file, step back 4kb (or something), and then
read forwards 4kb,
step back 8kb
repeat until pattern is found, making sure that handle reading partial lines correctly.
(This, apart from looking for a pattern, may actually be what tac does: one implementation of tac)
I have to deal with text files in a motley selection of formats. Here's an example (Columns A and B are tab delimited):
A B
a Name1=Val1, Name2=Val2, Name3=Val3
b Name1=Val4, Name3=Val5
c Name1=Val6, Name2=Val7, Name3=Val8
The files could have headers or not, have mixed delimiting schemes, have columns with name/value pairs as above etc.
I often have the ad-hoc need to extract data from such files in various ways. For example from the above data I might want the value associated with Name2 where it is present. i.e.
A B
a Val2
c Val7
What tools/techniques are there for performing such manipulations as one line commands, using the above as an example but extensible to other cases?
I don't like sed too much, but it works for such things:
var="Name2";sed -n "1p;s/\([^ ]*\) .*$var=\([^ ,]*\).*/\1 \2/p" < filename
Gives you:
A B
a Val2
c Val7
You have all the basic bash shell commands, for example grep, cut, sed and awk at your disposal. You can also use Perl or Ruby for more complex things.
From what I've seen I'd start with Awk for this sort of thing and then if you need something more complex, I'd progress to Python.
I would use sed:
# print section of file between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p' # case sensitive
Since you have cygwin, I'd go with Perl. It's the easiest to learn (check out the O'Reily book: Learning Perl) and widely applicable.
I would use Perl. Write a small module (or more than one) for dealing with the different formats. You could then run perl oneliners using that library. Example for what it would
look like as follows:
perl -e 'use Parser;' -e 'parser("in.input").get("Name2");'
Don't quote me on the syntax, but that's the general idea. Abstract the task at hand to allow you to think in terms of what you need to do, not how you need to do it. Ruby would be another option, it tends to have a cleaner syntax, but either language would work.