When input is read from terminal, GNU Parallel always displays a warning:
parallel: Warning: Input is read from the terminal. Only experts do this on purpose. Press CTRL-D to exit.
But sometimes I do want to read from terminal (e.g., when I'm copy & pasting stuff from elsewhere entry by entry). Is it possible to turn off this warning? I couldn't find such an option in man parallel or man parallel_tutorial.
Note that I don't want a cheap solution like 2>/dev/null, since warning messages from other programs will be turned off, too. For instance, consider the following simple script:
#!/bin/bash
function print12 () {
echo "printing $1 to stdout"
echo "printing $1 to stderr" >/dev/stderr
}
export -f print12
SHELL=/bin/bash parallel -k print12 2>/dev/null
Messages printed to stderr will all be suppressed.
Just realized that I can do a cat or some read </dev/tty to achieve my desired effect. But let's just focus on the original question.
It cannot be turned off. But see it as a praise: Since you are doing it on purpose, you are an expert (at least in the eyes of GNU Parallel).
As it is just a warning, you are free to paste your arguments and have them run: The warning does not stop GNU Parallel from reading your input.
If you really do not like the warning:
cat | parallel ...
Related
I have the script run_md.py which produces the file test.dcd from the input file named test.pdb.
I want to execute the same command on multiple input files (test*.pdb) on a remote server using GNU parallel and transfer the result back to the local computer. Therefore, I'm using the following command:
parallel --trc {.}.dcd -j 2 -S $SERVER1 './run_md.py {} 1000' ::: test*.pdb
The command is running as expected on the server using 2 slots. However, the files are not transferred back and I get the following error:
rsync: link_stat "/home/bougui/{.}.dcd" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [Receiver=3.1.1]
It looks like the replacement string is not working. How can I make it works?
Below is the output of parallel --version:
GNU parallel 20130922
Copyright (C) 2007,2008,2009,2010,2011,2012,2013 Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.
Web site: http://www.gnu.org/software/parallel
When using GNU Parallel for a publication please cite:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
What you are doing is 100% correct. So something on your system is breaking this. Please try this on another system and if possible follow REPORTING BUGS from man parallel.
The bug reported in that thread has been fixed and this feature works well with the latest version of GNU parallel (20160622). The GNU parallel version 20130922 packaged with Debian 8.5 is buggy for the usage of {.} string replacement, as described below:
With more test I found that the output file must be specified with a replacement string in the command run in parallel.
For testing purpose, you can find below a complete example that others can run:
echo This is input_file > input_file && parallel --trc {}.out -S $SERVER1 cat {} ">"{}.out ::: input_file
The example above works well. When I use the substitution string {.} as below:
echo This is input_file > input_file.in && parallel --trc {.}.out -S $SERVER1 cat {} ">"{.}.out ::: input_file
It works, as well. However, if I didn't specify {.}.out in the command run in parallel as below:
echo This is input_file > input_file.in && parallel --trc {.}.out -S $SERVER1 cat {} ">"input_file.out ::: input_file
... I reproduce the error:
rsync: link_stat "/home/bouvier/{.}.out" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [Receiver=3.1.1]
rsync: [Receiver] write error: Broken pipe (32)
Therefore the output file must be specified in the command run in parallel.
In ns-3 simulator documentation they provide a simple bash function to ease your life:
function waff {
CWD="$PWD"
cd $NS3DIR
./waf --cwd="$CWD" $*
cd -
}
This function is supposed to execute the ./waf program situated in the ns-3 root folder but inside the folder you are actually situated into.
So in the case of ~/project$ waff --run first waf will run the first script in the ~/project folder.
But if I try to run any simulation by adding one parameter to the script's command like ~/project$ waff --run "first --PrintHelp" it throws an error
waf: error: no such option: --PrintHelp.
It only works when I actually run the scripts from the root folder without the waff function.
How to modify the function to make it expand the $* to an argument between double commas?
Well, I feel embarrased because the solution was way easier than expected.
If anyone using DCE has the same problem, it's as easy as quoting the $*:
./waf --cwd="$CWD" $*
with:
./waf --cwd="$CWD" "$*"
This function works for me with bash (supposed you defined the environment variable $NS3DIR) :
function waff {
CWD="$PWD"
cd $NS3DIR >/dev/null
./waf --cwd="$CWD" "$#"
cd - >/dev/null
}
Proof it works is :
$ waff --run "wifi-simple-adhoc --help"
Waf: Entering directory `/home'
Waf: Leaving directory `/home'
'build' finished successfully (2.013s)
ns3.22-wifi-simple-adhoc-debug [Program Arguments] [General Arguments]
Program Arguments:
--phyMode: Wifi Phy mode [DsssRate1Mbps]
--rss: received signal strength [-80]
--packetSize: size of application packet sent [1000]
--numPackets: number of packets generated [1]
--interval: interval (seconds) between packets [1]
--verbose: turn on all WifiNetDevice log components [false]
General Arguments:
--PrintGlobals: Print the list of globals.
--PrintGroups: Print the list of groups.
--PrintGroup=[group]: Print all TypeIds of group.
--PrintTypeIds: Print all TypeIds.
--PrintAttributes=[typeid]: Print all attributes of typeid.
--PrintHelp: Print this help message.
$ waff --run wifi-simple-adhoc --command-template=" %s --help"
Waf: Entering directory `/home'
Waf: Leaving directory `/home'
'build' finished successfully (1.816s)
ns3.22-wifi-simple-adhoc-debug [Program Arguments] [General Arguments]
Program Arguments:
--phyMode: Wifi Phy mode [DsssRate1Mbps]
--rss: received signal strength [-80]
--packetSize: size of application packet sent [1000]
--numPackets: number of packets generated [1]
--interval: interval (seconds) between packets [1]
--verbose: turn on all WifiNetDevice log components [false]
General Arguments:
--PrintGlobals: Print the list of globals.
--PrintGroups: Print the list of groups.
--PrintGroup=[group]: Print all TypeIds of group.
--PrintTypeIds: Print all TypeIds.
--PrintAttributes=[typeid]: Print all attributes of typeid.
--PrintHelp: Print this help message.
I'm building a student database in Bourne Shell Script, and this is literally the very first time I've ever even seen code written like this, so I'm terribly out of my element. I need to make it so that when the user inputs a course, the program checks the user input vs a database of courses I already have, and if the course doesn't exist, promps the user to input a new course. This is what I'm trying:
echo "course-1: \c"
read course1
while [[ grep -i "$course1" course3.dat == 1]]
do
echo "course does not exist"
echo "course-1: \c"
read course1
done
echo "course-2: \c"
read course2
while [[ grep -i "$course2" course3.dat == 1]]
do
echo "course does not exist"
echo "course-2: \c"
read course2
done
But I'm getting errors "conditional binary operator expected" and "syntax error near `-i' ". I've been trying to google answers but I'm not coming up with anything useful. So I was wondering if any of you could help me? Thanks so much.
[[ is a shortcut for /bin/test, which isn't what you want. (Here's a man page about it.)
Try this instead:
while ! grep -i "$course1" course3.dat
Or
until grep -i "$course1" course3.dat
The grep expression evaluates to true when grep is successful (i.e. matching lines), and the ! inverts that. Until has built in the opposite semantics from while.
[[ and [ are "test", which is what you want.
However, different shells have different syntaxes; ksh or bash would interpret "[[" okay, but Bourne shell (normally /bin/sh) would not.
I have an ash script where I need to check whether the user has entered anything stupid. The proper use is:
script <read | monitor> -s <system | event> [-f filter] [-n number]
And I need to detect if user has for example evoked it like:
script read -s system dummydummy
In order to parse the proper arguments, I use
while getopts "s:f:n:" Option
do
case $Option in
s)logname=${OPTARG};;
f)filterspec=${OPTARG};;
n)numlines=${OPTARG};;
*)exit $E_OPTERROR;; # DEFAULT
esac
OPRIND=${OPTIND}
done
How can I detect if there are any leftover unparsed arguments?
You can use this after exiting the while-loop:
shift $(($OPTIND - 1))
echo "Remaining arguments: $#"
It works in bash-like shells, let us know if it works in ash too.
I want to filter the doxygen warnings with grep, to supress the undocumented parameter warning for certain parameters. I am trying this:
doxygen doxycfgfile | grep -v "parameter x"
however this seems to have absolutely no effect on the output. Neither the lines containing parameter x are suppressed nor all other lines. The output appears to be exactly the same.
I am using tcsh.
Presumably this is because the undocumented parameter warning messages are being written to standard error (stderr), rather than standard out (stdout). With the pipe (|) you are only piping stdout to grep's input.
You could try doing something like
doxygen doxycfgfile |& grep -v "parameter x"
From the advanced bash scripting guide:
If |& is used, the standard error of command1 is connected to command2's standard input through the pipe; it is shorthand for 2>&1 |.
Note, this was added in Bash 4, so for earlier versions you will have you use 2>&1 | in place of |&.
Alternatively, you could just get rid of the standard error output, using something like
doxygen doxycfgfile 2>/dev/null
This answer on askubuntu was the source for my answer.