I'm trying to understand what is happening behind the Rcpp::sourceCpp() call on a parallelized environment. Recently, this was partially addressed in the question: Using Rcpp function in parLapply on Windows.
Within this post, Dirk said,
"You need to run the sourceCpp() call in each spawned process, or else get them your code."
This was in response to questioner's use of distributing the Rcpp function to the worker processes. The questioner was sending the Rcpp function via:
clusterExport(cl = cl, varlist = "payoff")
I'm confused as to why this doesn't work. My thoughts are that this was what the objective of the clusterExport() is for.
The issue here is that the compiled code is not "exportable" to the spawned processes without being embedded in a package due to how binaries are linked into R's processes.
Traditionally, the clusterExport() statement allows for R specific code to be distributed to workers.
By using clusterExport() on an Rcpp function, you are only receiving the R declaration and not the underlying shared library. That is to say, the R CMD SHLIB given in Attributes.R is not shared with / exported to the workers. As a result, when a call is then made to an Rcpp function on the worker, R cannot find the correct shared library.
Take the previous question's function:
Rcpp::cppFunction("NumericVector payoff( double strike, NumericVector data) {
return pmax(data - strike, 0);
}")
Note: I'm using cppFunction() instead of sourceCpp() but the results are equivalent since cppFunction() calls sourceCpp() to create the function.
Typing the function name:
payoff
Yields the R declaration with a shared library pointer.
function (strike, data)
.Primitive(".Call")(<pointer: 0x1015ec130>, strike, data)
This shared library is only available on process that compiled the function.
Hence, why it is always ideal to embed compiled code within a package and then distribute the package.
Related
I have an issue working with DiagramBuilder and ManipulationStation classes.
It appears to me, that c++ API and the python bindings work differently in my case.
C++ API behaves as expected, while the python bindings result in the runtime error:
DiagramBuilder: Cannot operate on ports of System plant until it has been registered using AddSystem
How I use C++ API
In one of the ManipulationStation::Setup...() methods I inject a block of code, that adds an extra manipuland
const std::string sdf_path = FindResourceOrThrow("drake/examples/manipulation_station/models/bolt_n_nut.sdf");
RigidTransform<double> X_WC(RotationMatrix<double>::Identity(), Vector3d(0.0, -0.3, 0.1));
bolt_n_nut_ = internal::AddAndWeldModelFrom(sdf_path, "nut_and_bolt", lant_->world_frame(), "bolt", X_WC, plant_);
I inject another block of code into the method ManipulationStation::Finalize:
auto zero_torque = builder.template AddSystem<systems::ConstantVectorSource<double>>(Eigen::VectorXd::Zero(plant_->num_velocities(bolt_n_nut_)));
builder.Connect(zero_torque->get_output_port(), plant_->get_actuation_input_port(bolt_n_nut_));
With these changes, the simulation runs as expected.
How I use python bindings
plant = station.get_multibody_plant()
manipuland_path = get_manipuland_resource_path()
bolt_with_nut = Parser(plant=plant).AddModelFromFile(manipuland_path)
X_WC = RigidTransform(RotationMatrix.Identity(), [0.0, -0.3, 0.1])
plant.WeldFrames(plant.world_frame(), plant.GetFrameByName('bolt', bolt_with_nut), X_WC)
...
station.Finalize()
zero_torque = builder.AddSystem(ConstantValueSource(AbstractValue.Make([0.])))
builder.Connect(zero_torque.get_output_port(), plant.get_actuation_input_port(bolt_with_nut_model))
This triggers a RuntimeError with a message as above; The port, which causes this error is nut_and_bolt_actuation.
My vague understanding of the problem is the (in) visibility of nut_and_bolt System, due to having two distinct DiagramBuilders in a process: 1) a one is inside ManipulationStation 2) another is in the python code, that instantiates this ManipulationStation object.
Using ManipulationStation via python bindings is a preference for me, because that way I would've avoided depending on a custom build of drake library.
Thanks for your insight!
I agree with your assessment: you have two different DiagramBuilder objects here. This does not have anything to due with C++ or Python; the ManipulationStation is itself a Diagram (created using its own DiagramBuilder), and you have a second DiagramBuilder (in either c++ or python) that is connecting the ManipulationStation together with other elements. You are trying to connect a system that is in the external diagram to a port that is in the internal diagram, but is not exposed.
The solution would be to have the ManipulationStation diagram expose the extra nut and bolt actuation port so that you can connect to it from the second builder.
If you prefer Python, I've switched my course to using a completely python version of the manipulation station. I find this version is much easier to adapt to different student projects. (To be clear, the setup is in python, but at simulation time all of the elements are c++ and it doesn't call back to python; so the performance is almost identical.)
To simplify things, let's assume, that there are two Lua scripts, one program and one module, e.g:
sample.lua
local sample = {}
sample.fun1 = function()
end;
return sample;
I'm using it in program, like this:
program.lua
local sample = require("sample");
sample.fun1();
And now i'm trying to use the same module in C (first running lua program, what is important here), something like this:
luaL_dofile(luaState, "program.lua");
...
lua_getglobal(luaState, "sample");
lua_getfield(luaState,-1, "fun1");
or
luaL_dofile(luaState, "program.lua");
...
lua_getglobal(luaState, "sample.fun1");
But both versions causing crash. My question here: Is it possible to use somehow module, which was previously used in program? For this example it is of course nonsense, but my system is far more complicated and i need this functionality. Maybe some other approach to achieve this?
Question: Is there a way to call a function when the lua script is terminated by either the system or a program which has started the script (e.g. a C program)? An atexit()/cleaning function for lua.
The situation: a external C program (call it PROG) manages a lua script (call it SCRIPT) and calls its functions, the lua script uses a separated library (.so, cal it LIB) which reserves resources which should be freed when the lua script exits. The lua script is managed (and thus terminated) by PROG which i can not alter. SCRIPT should notify LIB when it is terminated.
How can this be done?
Note: i'm fairly new to lua so please also explain your answer, much appreciated :)
I'm on Linux using Lua 5.3.1
Currently this seem to work:
a = {__gc = function() print'exit function from LIB called' end}
setmetatable(a,a)
Check out http://lua-users.org/lists/lua-l/2001-08/msg00265.html
You may need to run lua_runprotected call to prevent a stack under flow problem.
I use VLC media player 1.1.9 on Ubuntu 11.04. I'm trying to experiment with lua extensions for VLC; so I've added the file test.lua in ~/.local/share/vlc/lua/extensions/, which has only these two lines:
fps="25.000"
frame_duration=1/tonumber(fps)
When I run vlc with verbose output for debugging, I get (edited to split on multiple lines:):
$ vlc --verbose 2
...
[0xa213874] lua generic warning: Error loading script
~/.local/share/vlc/lua/extensions/test.lua:
.../.local/share/vlc/lua/extensions/test.lua:2:
attempt to call global 'tonumber' (a nil value)
...
Now, as far as I know, tonumber as function is part of Lua5.1 proper (Lua 5.1 Reference Manual: tonumber) - and on my system:
$ locate --regex 'lua.*so.*' | head -4
/usr/lib/libipelua.so.7.0.10
/usr/lib/liblua5.1.so
/usr/lib/liblua5.1.so.0
/usr/lib/liblua5.1.so.0.0.0
... apparently I do have Lua 5.1 installed.
So, why do I get an error on using tonumber here - and how can I use this (and other) standard functions in a VLC lua extension properly?
Documentation is sparse for VLC Lua extensions to say the least but I did find an example in the github vlc repository here: https://github.com/videolan/vlc/blob/master/share/lua/extensions/VLSub.lua
Judging from that example it appears you need to supply some basic event functions for your addon for VLC to call into when certain events happen. Some of the obvious callback handlers I've noticed:
descriptor, this should return a table that contains fields describing your addon.
activate, this seems to get called when you activate it from view menubar.
deactivate, called when you deactivate the addon from view menubar.
plus a couple of other functions like close and input_change which you can guess what they're for.
From my brief testing done on VLC 2.0.8 under Win7 it appears VLC loads the lua extension using an empty sandbox environment. This is likely the reason you're getting nil for tonumber and I'm betting none of the other standard lua functions are accessible either when you try to perform computation at this global scope.
However, if I move that code into one of the event handling functions then all those standard functions are accessible again. For example:
function descriptor()
return
{
title = "Test Ext";
version = "0.1";
author = "";
shortdesc = "Testing Lua Extension";
capabilities = {};
description = "VLC Hello Test Addon";
}
end
function activate()
print "test activating"
local fps = tonumber "25.000"
local frame_duration = 1 / fps
print(frame_duration)
return true
end
-- ...
That prints out what you would expect in the console debug log. Now the documentation (what little there is) doesn't mention any of this but what's probably happening here is VLC is injecting the standard lua functions and vlc api table into the sandboxed environment when any of these event handlers get called. But during the extension loading phase, it is done in an empty sandbox environment which explains why all those lua function calls end up being nil when you try to use it at the outter most scope.
I recommend cloning the VLC source tree from github and then performing a grep on the C source that's embedding lua to see what VLC is really doing behind the scenes. Most of the relevant code will likely be here: https://github.com/videolan/vlc/tree/master/modules/lua
Probably some extension script installed in your system overwrites the function and the Lua interpreter instance is shared between all extension scripts, so you end up not being able to call the function if that script is called before yours.
As a quick workaround, Lua being dynamically typed, you can still do things like:
1 / "25.000"
and the string will be coerced to a number.
Alternatively, you can define a tonumber equivalent like:
string_to_num = function(s) return s + 0 end
This again relies on dynamic typing.
Mainly I want to know if I can send a function in a message in a distributed Erlang setup.
On Machine 1:
F1 = Fun()-> hey end,
gen_server:call(on_other_machine,F1)
On Machine 2:
handler_call(Function,From,State) ->
{reply,Function(),State)
Does it make sense?
Here's an interesting article about "passing fun's to other Erlang nodes". To resume it briefly:
[...] As you might know, Erlang distribution
works by sending the binary encoding
of terms; and so sending a fun is also
essentially done by encoding it using
erlang:term_to_binary/1; passing the
resulting binary to another node, and
then decoding it again using
erlang:binary_to_term/1.[...]
This is pretty obvious
for most data types; but how does it
work for function objects?
When you encode a fun, what is encoded
is just a reference to the function,
not the function implementation.
[...]
[...]the definition of the function is not passed along; just exactly enough information to recreate the fun at an other node if the module is there.
[...] If the module containing the fun has not yet been loaded, and the target node is running in interactive mode; then the module is attempted loaded using the regular module loading mechanism (contained in the module error_handler); and then it tries to see if a fun with the given id is available in said module. However, this only happens lazily when you try to apply the function.
[...] If you never attempt to apply the function, then nothing bad happens. The fun can be passed to another node (which has the module/fun in question) and then everybody is happy.
Maybe the target node has a module loaded of said name, but perhaps in a different version; which would then be very likely to have a different MD5 checksum, then you get the error badfun if you try to apply it.
I would suggest you to read the whole article, cause it's extremely interesting.
You can send any valid Erlang term. Although you have to be careful when sending funs. Any fun referencing a function inside a module needs that module to exist on the target node to work:
(first#host)9> rpc:call(second#host, erlang, apply,
[fun io:format/1, ["Hey!~n"]]).
Hey!
ok
(first#host)10> mymodule:func("Hey!~n").
5
(first#host)11> rpc:call(second#host, erlang, apply,
[fun mymodule:func/1, ["Hey!~n"]]).
{badrpc,{'EXIT',{undef,[{mymodule,func,["Hey!~n"]},
{rpc,'-handle_call_call/6-fun-0-',5}]}}}
In this example, io exists on both nodes and it works to send a function from io as a fun. However, mymodule exists only on the first node and the fun generates an undef exception when called on the other node.
As for anonymous functions, it seems they can be sent and work as expected.
t1#localhost:
(t1#localhost)7> register(shell, self()).
true
(t1#localhost)10> A = me, receive Fun when is_function(Fun) -> Fun(A) end.
hello me you
ok
t2#localhost:
(t2#localhost)11> B = you.
you
(t2#localhost)12> Fn2 = fun (A) -> io:format("hello ~p ~p~n", [A, B]) end.
#Fun<erl_eval.6.54118792>
(t2#localhost)13> {shell, 't1#localhost'} ! Fn2.
I am adding coverage logic to an app built on riak-core, and the merge of results gathered can be tricky if anonymous functions cannot be used in messages.
Also check out riak_kv/src/riak_kv_coverage_filter.erl
riak_kv might be using it to filter result, I guess.