I'm currently working with Julia (1.0) to run some parallel code on clusters of an HPC. The HPC is managed with PBS. I'm trying to find a way for broadcasting environment variables over all processes, i.e. a way to broadcast a specific list of environment variables automatically in order to have access to them in every Julia worker.
#!/bin/bash
#PBS ...
export TOTO=toto
julia --machine-file=$PBS_NODEFILE my_script.jl
In this example, I will not be able to access to the variable TOTO in each julia worker (via ENV["TOTO"]).
The only way I found to do what I want is to set the variables in my .bashrc but I want this to be script-specific. Another way is to put in my startup.jl file :
#everywhere ENV["TOTO"] = $(ENV["TOTO"])
But it is not script-specific because I have to know in advance which variables I want to send. If I do a loop over ENV keys then I'll broadcast all the variables and then override variables I don't want to.
I tried to use DotEnv.jl but it doesn't work.
Thanks for your time.
The obvious way is to set the variables first thing in script.jl. You can also put the initialization in a separate file, e.g. environment.jl, and load that on all processes with the -L flag:
julia --machine-file=$PBS_NODEFILE -L environment.jl my_script.jl
where environment.jl would, in this case, contain
ENV["TOTO"] = "toto"
etc.
Related
Is there any way to detect an environment variable change DURING the execution of a Tcl script (I use Tk so the execution can be long) ?
For instance, if I define an environment variable MYVAR=1, then I can access it from Tcl by writing $ENV(MYVAR). Let's say now that during the execution of the Tcl program, I switch MYVAR to 2. Is there a way, or maybe a command, that scans every environment variable again so I can get 2 when I call $ENV(MYVAR) ?
First off, other processes will not see changes to the environment variables of any process. Children get a copy of the current environment when they are created, and that's it.
Secondly, to see a change in the environment variables, put a trace on the ::env variable (but tracing an individual variable is not recommended). I can't remember if this works reliably between threads, but within a thread it's pretty good provided you don't have C code modifying the variables behind your back.
proc detectChange {name1 name2 op} {
# We know what name1 and op are in this case; I'll ignore them
if {$name2 eq "MYVAR"} {
puts "MYVAR changed to $::env(MYVAR)"
}
}
trace add variable ::env write detectChange
Note that Tk internally uses traces a lot (but the C API for them, not the Tcl language API for them).
ruby has some environment variables to configure garbage collection.
However, I do not know that the environment variables really have affected to ruby run-time.
How do I confirm them?
In my understanding, GC.stats show statistics but it does not show and correspond to environment variable values.
My purpose is to activate GC more frequently that means I want to reduce memory usage.
module GC
https://docs.ruby-lang.org/en/2.3.0/GC.html
e.g)
RUBY_GC_HEAP_INIT_SLOTS
RUBY_GC_HEAP_FREE_SLOTS
RUBY_GC_HEAP_GROWTH_FACTOR
RUBY_GC_HEAP_GROWTH_MAX_SLOTS
RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR
RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR
RUBY_GC_MALLOC_LIMIT
RUBY_GC_MALLOC_LIMIT_MAX
RUBY_GC_MALLOC_LIMIT_GROWTH_FACTOR
RUBY_GC_OLDMALLOC_LIMIT
RUBY_GC_OLDMALLOC_LIMIT_MAX
RUBY_GC_OLDMALLOC_LIMIT_GROWTH_FACTOR
You can inspect ENV in the running process.
Beyond that, I don't know how you would verify that the env vars you set actually did anything.
I'm trying to write a custom Travis env variable to a file for a simple proof of concept thing that I need. However, I'm having trouble getting this to work.
How would I define this in the travis yaml file if my variable is called VARIABLE_X ?
Thanks!
One way to do this is using linux commands, something like:
printenv | grep VARIABLE > all_env
However I don't know how Travis handles the environment (take a look at their docs, here) but it might not work as easily due to encryption, but it should work since your apps wouldn't function if they didn't have the same level of access. If such a case occurs, modifying a few parameters (maybe TRAVIS_SECURE_ENV_VARS) is worth looking into.
If you solved the problem in another way, consider sharing with the community.
Write the environment variable as usual (Shell - Write variable contents to a file)
Define the following within script:
- echo "$VARIABLE_X" > example.txt
I have a question about following lines related to adding PATH to enviroment.
export PATH=/usr/loca/cuda/bin:$PATH
export PATH=/usr/local/cuda-9.1/bin${PATH:+:${PATH}}
export PATH="/home/ics_vr/anaconda3/bin:$PATH"
export PATH="$PATH:/home/user/anaconda3/bin"
Regardless the content of path in each export lines, my first question is how do I distinguish thoes lines starting with export PATH=? e.g. grammer and its functions, regardless the variable I used in thoes lines.
Secondly, I see many people use # to comment on/off to switch those path,but this is not convenient. Is there any union way to realize all, without commenting the export line every time?
This is convenient because people want to use system python for example as default, but if the path is settled not properly, anaconda python interpreter will be settled by default. We need a way that default is the system python interpreter, and when I need anaconda, I will use
source activate ENV_I_BUILD
Thank you for your time and help. I am very appreciate on that.
The environment variable PATH is a list of colon separated folder paths where to find executables.
The order in which folder paths are places in this variable is very important. Indeed, if you call a program from the command line, the executable will first be searched in the first folder path, then if it's not there the second and so on...
Anaconda ships with a python installation (either 2.x or 3.x).
If you export:
export PATH="/home/ics_vr/anaconda3/bin:$PATH"
then the python in "/home/ics_vr/anaconda3/bin/anaconda3" will be used preferentially. Thus, if you want to keep the system python by default, you might want to use:
export PATH="$PATH:/path/to/whatever/conda"
The source activate ... will prepend the environment bin folder in your PATH anyway. So if you activate an environment, the system python will be superseeded by the python of the conda env.
As for the two lines:
export PATH=/usr/loca/cuda/bin:$PATH
export PATH=/usr/local/cuda-9.1/bin${PATH:+:${PATH}}
you will have to decide what executables you want first in your PATH variable.
For information, you can set multiple folders in your PATH in one line:
export PATH="$PATH:/usr/loca/cuda/bin:/home/ics_vr/anaconda3/bin:/my/personal/bin"
Do not forget to add what was already in your PATH variable when exporting a new PATH if you do not want to loose basic commands listed in, for example, "/usr/bin" or "/usr/local/bin".
It is my understanding that when invoking spawn "string command" in xmonad, the argument "string command" is actually passed to /bin/sh.
Is there a way to change this behavior ?
More specifically, is it possible to make the instance of the interpreter called by spawn aware of some predefined environment variables (typically, SSH_AUTH_SOCK and SSH_AGENT_PID)?
Of course, it is always possible to resort to spawn "$VARIABLE=stuff; export $VARIABLE; string command", but it bothers me that the variabe should be created and exported each time.
Strictly answering your first question, the safeSpawn function in XMonad.Util.Run (in xmonad-contrib) will run a command without passing it to a shell.
However, that shouldn't make much of a difference as far as environment variables are concerned. In both cases, the spawned command should inherit the environment of the XMonad process (which the shell's startup/rc files could tweak in the case of spawn).
It's possible to set the environment of the started process with general Haskell facilities, e.g. System.Posix.Process.executeFile (and System.Environment.getEnvironment if you want to make a modified copy of the XMonad process' environment).