I would like to test (from xonsh) if a command is available or not. If I try this from the xonsh command prompt:
which bash
Then it works:
user#server ~ $ which bash
/usr/bin/bash
But it does not work from xonsh script:
#!/usr/bin/env xonsh
$RAISE_SUBPROC_ERROR = True
try:
which bash
print("bash is available")
except:
print("bash is not available")
Because it results in this error:
NameError: name 'which' is not defined
I understand that which is a shell builtin. E.g. it is not an executable file. But it is available at the xnosh command prompt. Then why it is not available inside an xonsh script? The ultimate question is this: how can I test (from an xonsh script) if a (subprocess mode) command is available or not?
import shutil
print(shutil.which('bash'))
While nagylzs' answer led me to the right solution, I found it inadequate.
shutil.which defaults to os.environ['PATH']. On my machine, the default os.environ['PATH'] doesn't contain the active PATH recognized by xonsh.
~ $ os.environ['PATH']
'/usr/bin:/bin:/usr/sbin:/sbin'
I found I needed to pass $PATH to reliably resolve 'which' in the xonsh environment.
~ $ $PATH[:2]
['/opt/google-cloud-sdk/bin', '/Users/jaraco/.local/bin']
~ $ import shutil
~ $ shutil.which('brew', path=os.pathsep.join($PATH))
'/opt/homebrew/bin/brew'
The latest version of xonsh includes a built-in which command. Unfortunately, the version included will emit an error on stdout if the target isn't found, a behavior that is not great for non-interactive use.
As mentioned in another answer, which exists in the current version of xonsh (0.13.4 as of 15/12/2022) so your script would work. However, it outputs its own error message so it's necessary to redirect stderr to get rid of it.
Also, unless you redirect its stdout as well (using all>), it migh be a good idea to capture its output so the final version would look like this:
#!/usr/bin/env xonsh
$RAISE_SUBPROC_ERROR = True
try:
bash = $(which bash err> /dev/null)
print(f"bash is available: {bash}")
except:
print("bash is not available")
Related
When I use any command with sudo the environment variables are not there. For example after setting HTTP_PROXY the command wget works fine without sudo. However if I type sudo wget it says it can't bypass the proxy setting.
First you need to export HTTP_PROXY. Second, you need to read man sudo, and look at the -E flag. This works:
$ export HTTP_PROXY=foof
$ sudo -E bash -c 'echo $HTTP_PROXY'
Here is the quote from the man page:
-E, --preserve-env
Indicates to the security policy that the user wishes to preserve their
existing environment variables. The security policy may return an error
if the user does not have permission to preserve the environment.
The trick is to add environment variables to sudoers file via sudo visudo command and add these lines:
Defaults env_keep += "ftp_proxy http_proxy https_proxy no_proxy"
taken from ArchLinux wiki.
For Ubuntu 14, you need to specify in separate lines as it returns the errors for multi-variable lines:
Defaults env_keep += "http_proxy"
Defaults env_keep += "https_proxy"
Defaults env_keep += "HTTP_PROXY"
Defaults env_keep += "HTTPS_PROXY"
For individual variables you want to make available on a one off basis you can make it part of the command.
sudo http_proxy=$http_proxy wget "http://stackoverflow.com"
You can also combine the two env_keep statements in Ahmed Aswani's answer into a single statement like this:
Defaults env_keep += "http_proxy https_proxy"
You should also consider specifying env_keep for only a single command like this:
Defaults!/bin/[your_command] env_keep += "http_proxy https_proxy"
A simple wrapper function (or in-line for loop)
I came up with a unique solution because:
sudo -E "$#" was leaking variables that was causing problems for my command
sudo VAR1="$VAR1" ... VAR42="$VAR42" "$#" was long and ugly in my case
demo.sh
#!/bin/bash
function sudo_exports(){
eval sudo $(for x in $_EXPORTS; do printf '%q=%q ' "$x" "${!x}"; done;) "$#"
}
# create a test script to call as sudo
echo 'echo Forty-Two is $VAR42' > sudo_test.sh
chmod +x sudo_test.sh
export VAR42="The Answer to the Ultimate Question of Life, The Universe, and Everything."
export _EXPORTS="_EXPORTS VAR1 VAR2 VAR3 VAR4 VAR5 VAR6 VAR7 VAR8 VAR9 VAR10 VAR11 VAR12 VAR13 VAR14 VAR15 VAR16 VAR17 VAR18 VAR19 VAR20 VAR21 VAR22 VAR23 VAR24 VAR25 VAR26 VAR27 VAR28 VAR29 VAR30 VAR31 VAR32 VAR33 VAR34 VAR35 VAR36 VAR37 VAR38 VAR39 VAR40 VAR41 VAR42"
# clean function style
sudo_exports ./sudo_test.sh
# or just use the content of the function
eval sudo $(for x in $_EXPORTS; do printf '%q=%q ' "$x" "${!x}"; done;) ./sudo_test.sh
Result
$ ./demo.sh
Forty-Two is The Answer to the Ultimate Question of Life, The Universe, and Everything.
Forty-Two is The Answer to the Ultimate Question of Life, The Universe, and Everything.
How?
This is made possible by a feature of the bash builtin printf. The %q produces a shell quoted string. Unlike the parameter expansion in bash 4.4, this works in bash versions < 4.0
Add code snippets to /etc/sudoers.d
Don't know if this is available in all distros, but in Debian-based distros, there is a line at or near the tail of the /etc/sudoers file that includes the folder /etc/sudoers.d. Herein, one may add code "snippets" that modify sudo's configuration. Specifically, they allow control over all environment variables used in sudo.
As with /etc/sudoers, these "code snippets" should be edited using visudo. You can start by reading the README file, which is also a handy place for keeping any notes you care to make:
$ sudo visudo -f /etc/sudoers.d/README
# files for your snippets may be created/edited like so:
$ sudo visudo -f /etc/sudoers.d/20_mysnippets
Read the "Command Environment" section of 'man 5 sudoers'
Perhaps the most informative documentation on environment configuration in sudo is found in the Command environment section of man 5 sudoers. Here, we learn that a sudoers environment variables that are blocked by default may be "whitelisted" using the env_check or env_keep options; e.g.
Defaults env_keep += "http_proxy HTTP_PROXY"
Defaults env_keep += "https_proxy HTTPS_PROXY"
Defaults env_keep += "ftp_proxy FTP_PROXY"
And so, in the OP's case, we may "pass" the sudoer's environment variables as follows:
$ sudo visudo -f /etc/sudoers.d/10_myenvwlist
# opens the default editor for entry of the following lines:
Defaults env_keep += "http_proxy HTTP_PROXY"
Defaults env_keep += "https_proxy HTTPS_PROXY"
# and any others deemed useful/necessary
# Save the file, close the editor, and you are done!
Get your bearings from '# sudo -V'
The OP presumably discovered the missing environment variable in sudo by trial-and-error. However, it is possible to be proactive: A listing of all environment variables, and their allowed or denied status is available (and unique to each host) from the root prompt as follows:
# sudo -V
...
Environment variables to check for safety:
...
Environment variables to remove:
...
Environment variables to preserve:
...
Note that once an environment variable is "whitelisted" as above, it will appear in subsequent listings of sudo -V under the "preserve" listing.
If you have the need to keep the environment variables in a script you can put your command in a here document like this. Especially if you have lots of variables to set things look tidy this way.
# prepare a script e.g. for running maven
runmaven=/tmp/runmaven$$
# create the script with a here document
cat << EOF > $runmaven
#!/bin/bash
# run the maven clean with environment variables set
export ANT_HOME=/usr/share/ant
export MAKEFLAGS=-j4
mvn clean install
EOF
# make the script executable
chmod +x $runmaven
# run it
sudo $runmaven
# remove it or comment out to keep
rm $runmaven
I'm running a snakemake pipeline that for a specific rule loads a container:
rule counts:
params:
transcriptome=os.environ["INDEX"],
outdir= (os.environ["OUTDIR"] + "/counts/"),
indir= (os.environ["INDIR"] + "{sample}"),
name = lambda wildcards: SAMPLES[wildcards.sample]
output:
(os.environ["OUTDIR"] + "counts/" + "{sample}" + "/outs/web_summary.html")
container:
"docker://marcusczi/cellranger_clean"
shell:
"""
cellranger count --id={wildcards.sample} --transcriptome={params.transcriptome} --fastqs={params.indir} --sample={params.name}
mkdir -p {params.outdir}
mv ./{wildcards.sample}/ {params.outdir}
"""
Dry run looks fine, the rule itself I'm sure it works (tried it without the container). However, when I run it with docker I get this error:
Activating singularity image /some/path/.snakemake/singularity/c288fbc3fef5771f055a688c6678c24d.simg
/bin/sh: syntax error: unterminated quoted string
[ 1.228141] reboot: Power down
And then it waits for the missing files, and fails.
I think the answer to this situation might be related to this previous question, but I have tried everything i can think of in terms of escaping characters (except for the wildcards and variables within curly brackets because I'm guessing it should be fine, and if not why am i even using snakemake :-( ). The paths for the directories I'm using are valid and exist, the name and wildcard "sample" are in the shape "sample_123", nothing fancy.
It's also worth saying that there are no single or double quotes in any of these variables.
Thank you!!
Software and OS:
I am in macos catalina 10.15.5, running snakemake 5.20.1, and I have been using the beta version of singularity for macos (3.3.0-rc.1.658.g7427b73f1.dirty).
Running singularity outside Snakemake:
I tried running the singularity outside snakemake, the software that I'm trying to run starts, but then complains that there is no disk left on space (which is not true). I'm running the singularity as sudo singularity run -B "$(pwd):$(pwd)" docker://marcusczi/cellranger_clean
I think this latest error might be either 1) I'm not running singularity as I should..? Or 2) A false statement of what is happening since cellranger (the software I'm trying to run) often has misleading error messages.
Minimal reproducible example:
If you install snakemake, you should be able to reproduce my error when running snakemake -j1 --use-singularity in the same directory of the Snakefile.
Snakefile:
rule all:
input:
"output.txt"
rule counts:
output:
"output.txt"
container:
"docker://marcusczi/cellranger_clean"
shell:
"""
cellranger count --help
echo "hurray!" > {output}
"""
I'm trying to determine the location of the 'flutter' script on a user's computer using this Dart code:
Process.run('which', ['flutter'], runInShell: true).then((results) {
print('which returned code ${results.exitCode}\n StdOut: ${results.stdout}\n StdErr: ${results.stderr}');
}
The problem is that the PATH environment has been set in the user's .bash_profile file like this:
export PATH=$PATH:/Users/mr_pink/dev/flutter/bin
but apparently the .bash_profile script doesn't get loaded when I call Process.run. How can I make sure it is loaded so that the "which flutter" command uses the correct PATH variable?
After reading man bash I found the solution was to add -l to the bash command like this:
Process.run('bash', ['-l', '-c', 'which flutter'])
I'm trying to learn to write Nix expressions, and I thought about doing my own very simple "Hello World!" (as is tradition).
So I have my dir with only this default.nix file :
{pkgs ? import <nixpkgs> {}}:
derivation {
system = "x86_64-linux";
name = "simple-bash-derivation-helloworld";
builder = pkgs.bash;
args = [ "-c" "echo 'Hello World' > $out" ];
}
Here is what I get when I try to build it:
nix-build
these derivations will be built:
/nix/store/3grmahx3ih4c50asj84p7xnpqpj32n5s-simple-bash-derivation-helloworld.drv
building path(s) ‘/nix/store/6psl3rc92311w37c1n6nj0a6jac16hv1-simple-bash-derivation-helloworld’
while setting up the build environment: executing ‘/nix/store/wb34dgkpmnssjkq7yj4qbjqxpnapq0lw-bash-4.4-p12’: Permission denied
builder for ‘/nix/store/3grmahx3ih4c50asj84p7xnpqpj32n5s-simple-bash-derivation-helloworld.drv’ failed with exit code 1
error: build of ‘/nix/store/3grmahx3ih4c50asj84p7xnpqpj32n5s-simple-bash-derivation-helloworld.drv’ failed
Removing the args line yields the same issue.
Why do I get a permission issue?
What would be the correct way to make a simple derivation just doing a bash echo?
Please note that this is a learning exercise: I do not want to use stdenv.mkDerivation here for example.
I am running nix-env (Nix) 1.11.9 on an Ubuntu 16.04 system.
Thanks in advance.
Try running ls command on /nix/store/wb34dgkpmnssjkq7yj4qbjqxpnapq0lw-bash-4.4-p12 and you will see it's a directory rather than an executable file (pointing to $out of the pkgs.bash derivation). If you wanted to refer to bash binary you would use:
builder = "${pkgs.bash}/bin/bash";
How do I run a Windows command in an Ruby app?
I am trying to run something like:
output = `cd #{RAILS_ROOT}/lib && java HelloWorld #{param1} #{param2}`
I print the result of the line above and paste it to a command prompt in Windows and it works just fine. However, when i run app and hit this code, output is blank rather than have a string I get back from HellowWorld. In HelloWorld I do a System.out.print("helloworld")
The following:
output = `cmd.exe /C dir`
puts "OUTPUT #{output}"
Returns:
OUTPUT
Issue in JRuby 1.5.3 fixed in JRuby 1.5.5:
http://www.jruby.org/2010/11/10/jruby-1-5-5.html
Try to use File#join here. It will generate crossplatform path for you
http://apidock.com/ruby/File/join/class
my_path = File.join(RAILS_ROOT, "lib")
output = `cd #{my_path} && java HelloWorld #{param1} #{param2}`
Also you can execute your system commands this way:
`cd #{my_path} && java HelloWorld #{param1} #{param2}`
system("cd #{my_path} && java HelloWorld #{param1} #{param2}")
%x[cd #{my_path} && java HelloWorld #{param1} #{param2}]
Related topic: System call from Ruby
Backticks work fine for me. Try:
output = `dir`
to prove to yourself that it's working. At that point, your question is how to run a Java app from the command line, or why your particular app isn't work. Note that you can temporarily change the working directory like this:
Dir.chdir(File.join(RAILS_ROOT,'lib')) do
output = `...`
end