I am trying to chain a few commands with gnu-parallel. Those commands involve environment variables defined before. For instance, consider the command below, which receives a list of *.c files, emit llvm bytecode with clang to .bc files and optimize it into .rbc files with opt.
parallel --halt now,fail=1 'clang -Xclang -disable-O0-optnone -S -c -emit-llvm {} -o {.}.bc ; opt -S -mem2reg {.}.bc -o {.}.rbc ' ::: "${source_files[#]}"
The program above works just fine, but if I change clang by an ambient variable, the command stops working.
parallel --halt now,fail=1 '$COMPILER -Xclang ... ' ::: "${source_files[#]}"
The shell does not expand environment variables inside single quotes, and parallel does not do so either (hence the env_parallel suggestion in the comments).
The easiest solution here is to use different quotes around the variable:
parallel --halt now,fail=1 "$COMPILER"' -Xclang ... ' ::: "${source_files[#]}"
I've left the single quotes around the rest of the command, just in case, but you may not need to. Quotes placed back-to-back like that are automatically concatenated into one argument by the shell.
Related
I am working on LLVM obfuscation project. I have written a llvm pass(lets say flow flattening pass) which i am running on source (test.c) with following command:
clang -emit-llvm test.c -c -o test.bc
opt -load ../../.. LLVMFlattening.so -fla <test.bc>/dev/null
But i have seen that in O-LLVM project they achieved same thing using:
clang -emit-llvm test.c -c -o test.bc -mllvm -fla
Can someone tell me what is -mllvm here and how this changed to a simple command?
-mllvm means Additional arguments to forward to LLVM's option processing. Therefore -mllvm -fla will pass -fla to the LLVM's option processing.
Clang and LLVM could run seperately. If you want clang to run llvm, and also have some options which you want llvm to aware. -mllvm is what you need.
Defautly, LLVM does not turn on all the transformation passes. With -fla, LLVM will turn on the pass registered with command line argument fla by call function RegisterPass<typename passName>.
In your command line, opt's -load option is used to load plugin. If you want to use the simple command line as expect. Your pass need to be linked into the opt binary. This could be done in the following two ways:
(Without modify the existing LLVM source tree): Add your only pass's source by adding CMakeLists.txt mentioned in this link
Directly copy your pass source code folder into <LLVM root>/lib/Transform directory. And modify the <LLVM root>/lib/Transform/CMakeLists.txt, add add_subdirectory(<pass name>) line just like others.
I'm working on O-LLVM rencently, and came into the same problem. Here is my solution:
1.add static cl::opt<bool> YOUR_FLA("fla", cl::init(false),"info...") to PassManagerBuilder.cpp
2.add function Pass *createYOUR_FLA(bool flag) in your obfuscation pass source code
3.add MPM.add(createYOUR_FLA(YOUR_FLA)); to function populateModulePassManager in PassManagerBuilder.cpp
The solution above works with my simple pass.
Using clang-check to dump a source code's AST, can be done with the following command:
$ clang-check -ast-dump file.c --
However, the output of this command will appear colorful within the terminal.
When I direct the output to a file, I'm stuck with all of the color escape codes:
$ clang-check -ast-dump file.c -- > out.txt
example:
[0;1;32mTranslationUnitDecl[0m[0;33m 0x227c5c0[0m <[0;33m<invalid sloc>[0m> [0;33m<invalid sloc>[0m
[0;34m|-[0m[0;1;32mTypedefDecl[0m[0;33m 0x227cac0[0m <[0;33m<invalid sloc>[0m> [0;33m<invalid sloc>[0m implicit[0;1;36m __int128_t[0m [0;32m'__int128'[0m
[0;34m|-[0m[0;1;32mTypedefDecl[0m[0;33m 0x227cb20[0m <[0;33m<invalid sloc>[0m> [0;33m<invalid sloc>[0m implicit[0;1;36m __uint128_t[0m [0;32m'unsigned __int128'[0m
[0;34m|-[0m[0;1;32mTypedefDecl[0m[0;33m 0x227ce70[0m <[0;33m<invalid sloc>[0m> [0;33m<invalid sloc>[0m implicit[0;1;36m __builtin_va_list[0m [0;32m'__va_list_tag [1]'[0m
...
Is there a flag to disable colors in clang-check?
I tried adding the following flag, but it did not work:
--extra-arg="--no-color-diagnostics"
You are almost correct. Try
$ clang-check -ast-dump test.c --extra-arg="-fno-color-diagnostics" --
Additionally, -fno-diagnostics-color and -fdiagnostics-color=never also seems to work
Reference: http://clang.llvm.org/docs/UsersManual.html#formatting-of-diagnostics
When input is read from terminal, GNU Parallel always displays a warning:
parallel: Warning: Input is read from the terminal. Only experts do this on purpose. Press CTRL-D to exit.
But sometimes I do want to read from terminal (e.g., when I'm copy & pasting stuff from elsewhere entry by entry). Is it possible to turn off this warning? I couldn't find such an option in man parallel or man parallel_tutorial.
Note that I don't want a cheap solution like 2>/dev/null, since warning messages from other programs will be turned off, too. For instance, consider the following simple script:
#!/bin/bash
function print12 () {
echo "printing $1 to stdout"
echo "printing $1 to stderr" >/dev/stderr
}
export -f print12
SHELL=/bin/bash parallel -k print12 2>/dev/null
Messages printed to stderr will all be suppressed.
Just realized that I can do a cat or some read </dev/tty to achieve my desired effect. But let's just focus on the original question.
It cannot be turned off. But see it as a praise: Since you are doing it on purpose, you are an expert (at least in the eyes of GNU Parallel).
As it is just a warning, you are free to paste your arguments and have them run: The warning does not stop GNU Parallel from reading your input.
If you really do not like the warning:
cat | parallel ...
I just discovered GNU parallel and I'm having some trouble running a simple parallel task. I have a simulation running over multiple values and I'd like to split it up to run in parallel using command line args. From the docs , it seems you can run parallel mycommand :::: myargfile in which myargfile contains the various arguments you would like to feed your command, in parallel. However, I didn't see any information on how the args should be listed and assumed a myargfile like this would work:
--pmin 0 --pmax 0.1
--pmin 0.1 --pmax 0.2
...
mycommand --pmin 0 --pmax 0.1 executes no problem. But when I run parallel mycommand :::: myargfile I get error: unknown option pmin 0 --pmax 0.1 (caught and decoded courtesy boost program options). parallel echo :::: myargfile correctly prints out the arguments. It's as if they are being wrapped in a string which the program can't read and not fed like they are from a standard bash script.
What's going on? How can I make this work?
Following #DmitriChubarov's link to https://stackoverflow.com/a/6258206/1328439 , I discovered that I was lacking the colsep flag:
parallel --colsep ' ' mycommand :::: myargfile
successfully executes.
After digging through manual and help pages I came up with this example. Perhaps it will save someone out there. :)
#!/usr/bin/env bash
COMMANDS=(
"cnn -a mode=flat"
"cnn -a mode=xxx"
"cnn_x -a mode=extreme"
)
parallel --verbose --progress --colsep ' ' scrapy crawl {.} ::: "${COMMANDS[#]}"
I'm trying to set up a shell script that will start a screen session (or rejoin an existing one) only if it is invoked from an interactive shell. The solution I have seen is to check if $- contains the letter "i":
#!/bin/sh -e
echo "Testing interactivity..."
echo 'Current value of $- = '"$-"
if [ `echo \$- | grep -qs i` ]; then
echo interactive;
else
echo noninteractive;
fi
However, this fails, because the script is run by a new noninteractive shell, invoked as a result of the #!/bin/sh at the top. If I source the script instead of running it, it works as desired, but that's an ugly hack. I'd rather have it work when I run it.
So how can I test for interactivity within a script?
Give this a try and see if it does what you're looking for:
#!/bin/sh
if [ $_ != $0 ]
then
echo interactive;
else
echo noninteractive;
fi
The underscore ($_) expands to the absolute pathname used to invoke the script. The zero ($0) expands to the name of the script. If they're different then the script was invoked from an interactive shell. In Bash, subsequent expansion of $_ gives the expanded argument to the previous command (it might be a good idea to save the value of $_ in another variable in order to preserve it).
From man bash:
0 Expands to the name of the shell or shell script. This is set
at shell initialization. If bash is invoked with a file of com‐
mands, $0 is set to the name of that file. If bash is started
with the -c option, then $0 is set to the first argument after
the string to be executed, if one is present. Otherwise, it is
set to the file name used to invoke bash, as given by argument
zero.
_ At shell startup, set to the absolute pathname used to invoke
the shell or shell script being executed as passed in the envi‐
ronment or argument list. Subsequently, expands to the last
argument to the previous command, after expansion. Also set to
the full pathname used to invoke each command executed and
placed in the environment exported to that command. When check‐
ing mail, this parameter holds the name of the mail file cur‐
rently being checked.
$_ may not work in every POSIX compatible sh, although it probably works in must.
$PS1 will only be set if the shell is interactive. So this should work:
if [ -z "$PS1" ]; then
echo noninteractive
else
echo interactive
fi
try tty
if tty 2>&1 |grep not ; then echo "Not a tty"; else echo "a tty"; fi
man tty :
The tty utility writes the name of the terminal attached to standard
input to standard output. The name that is written is the string
returned by ttyname(3). If the standard input is not a terminal, the
message ``not a tty'' is written.
You could try using something like...
if [[ -t 0 ]]
then
echo "Interactive...say something!"
read line
echo $line
else
echo "Not Interactive"
fi
The "-t" switch in the test field checks if the file descriptor given matches a terminal (you could also do this to stop the program if the output was going to be printed to a terminal, for example). Here it checks if the standard in of the program matches a terminal.
Simple answer: don't run those commands inside ` ` or [ ].
There is no need for either of those constructs here.
Obviously I can't be sure what you expected
[ `echo \$- | grep -qs i` ]
to be testing, but I don't think it's testing what you think it's testing.
That code will do the following:
Run echo \$- | grep -qs i inside a subshell (due to the ` `).
Capture the subshell's standard output.
Replace the original ` ` expression with a string containing that output.
Pass that string as an argument to the [ command or built-in (depending on your shell).
Produce a successful return code from [ only if that string was nonempty (assuming the string didn't look like an option to [).
Some possible problems:
The -qs options to grep should cause it to produce no output, so I'd expect [ to be testing an empty string regardless of what $- looks like.
It's also possible that the backslash is escaping the dollar sign and causing a literal 'dollar minus' (rather than the contents of a variable) to be sent to grep.
On the other hand, if you removed the [ and backticks and instead said
if echo "$-" | grep -qs i ; then
then:
your current shell would expand "$-" with the value you want to test,
echo ... | would send that to grep on its standard input,
grep would return a successful return code when that input contained the letter i,
grep would print no output, due to the -qs flags, and
the if statement would use grep's return code to decide which branch to take.
Also:
no backticks would replace any commands with the output produced when they were run, and
no [ command would try to replace the return code of grep with some return code that it had tried to reconstruct by itself from the output produced by grep.
For more on how to use the if command, see this section of the excellent BashGuide.
If you want to test the value of $- without forking an external process (e.g. grep) then you can use the following technique:
if [ "${-%i*}" != "$-" ]
then
echo Interactive shell
else
echo Not an interactive shell
fi
This deletes any match for i* from the value of $- then checks to see if this made any difference.
(The ${parameter/from/to} construct (e.g. [ "${-//[!i]/}" = "i" ] is true iff interactive) can be used in Bash scripts but is not present in Dash, which is /bin/sh on Debian and Ubuntu systems.)