Passing strings to .wasm module - memory

I've been stuck on this for a while now and I cannot seem to find good resources to my problem. I am coming from and "only C" background, so most of the web dev stuff is completely new for me.
I wrote a C function float editDistance(char *str1, char *str2) that returns the edit distance of 2 char arrays. Right now the goal is to successfully call this function from a JS environment.
After ensuring that the code works with the recommended Emscipten ccall method, I decided to move on. Now
I use Emscripten to compile the C code with flags -O3, -s WASM=1, -s EXPORTED_FUNCTIONS="['_editDistance']", and -s SIDE_MODULE=1 -s to Wasm. The JS code I'm trying to wrap around my WebAssembly is:
// Allocate memory for the wasm module to run in. (65536*256 bit)
let wasmMemory = new WebAssembly.Memory({
initial: 256
});
let info = {
env: {
abort: function() {},
memoryBase: 0,
tableBase: 0,
memory: wasmMemory,
table: new WebAssembly.Table({initial: 2, element: 'anyfunc'}),
}
}
// Define the strings
let str1 = "abcd";
let str2 = "abcd";
// Allocate memory on the wasm partition for the HEAPU8
let HEAPU8 = new Uint8Array(wasmMemory.buffer);
// Create the char arrays on the heap from the strings
let stackPtr = 0;
let str1Ptr = stackPtr;
stackPtr = stringToASCIIArray(str1, HEAPU8, stackPtr);
let str2Ptr = stackPtr;
stackPtr = stringToASCIIArray(str2, HEAPU8, stackPtr);
// Read the wasm file and instantiate it with the above environment setup. Then
// call the exported function with the string pointers.
let wasmBinaryFile = 'bin/edit_distanceW.wasm';
fetch(wasmBinaryFile, {credentials:"same-origin"})
.then((response) => response.arrayBuffer())
.then((binary) => WebAssembly.instantiate(binary,info))
.then((wa) => alert(wa.instance.exports._editDistance(str1Ptr, str2Ptr)));
// Converts a string to an ASCII byte array on the specified memory
function stringToASCIIArray(str, outU8Array, idx){
let length = str.length + 1;
let i;
for(i=0; i<length; i++){
outU8Array[idx+i] = str.charCodeAt(i);
}
outU8Array[idx+i]=0;
return (idx + length);
}
The generated wasm file when converted to wat demands these imports:
(import "env" "abort" (func (;0;) (type 0)))
(import "env" "memoryBase" (global (;0;) i32))
(import "env" "tableBase" (global (;1;) i32))
(import "env" "memory" (memory (;0;) 256))
(import "env" "table" (table (;0;) 2 anyfunc))
.. and exports these:
(export "__post_instantiate" (func 7))
(export "_editDistance" (func 9))
(export "runPostSets" (func 6))
(elem (;0;) (get_global 1) 8 1))
Now, when I test the code the strings are passed to the C module without a problem. A few function calls are even made on them (strLen) before things go south. In the C function there is this nasty nested loop that does the main computation, iterating thru a 2D array while reading the characters from the strings (C code just been ported from a paper with an ugly pseudo code, so pardon me the variable names):
do{
for(p=0; p<editDistance; p++){
// Do stuff
}
// Do more stuff
editDistance++;
} while(fkp[len2*2-len1][editDistance] != len1);
Before the function enters the for() loop, the module still has the strings on memory str1Ptr=0x00 and str2Ptr=0x05 with the correct length and content. On the contrary, immediately after entering the for() loop the memory gets overwritten by garbage (mostly 0s), corrupting the end result. I suspect some stack saving and restoration problems on the scope change, as the exact same code compiled to my PC using gcc works like a charm.
Any idea what setup I'm missing that hinders the correct completion of the C function?

If you are starting out you probably want to use the emscripten-generated JS glue. That is, don't use SIDE_MODULE=1 and instead output to a files calle .js. The emscripten compiler will then generate both a .js and a .wasm file. You can then include the .js file in your project and it will handle all the loading and setup for you.
If you try to load the wasm file yourself, you will need to do a lot of work to replicate the emscripten environment, which will require a lot of internal details of emscripten. Also, those internal details of subject to change when you update to the new version of emscripten so you are creating more work for yourself.

Related

LuaJ does not supply command line arguments correctly

I tried the utility method provided by luaj to call a lua file with command line args (this one http://lua-users.org/wiki/SourceCodeFormatter)
Globals globals = JsePlatform.standardGlobals();
String script ="src/codeformatter.lua";
File f = new File(script);
LuaValue chunk = globals.loadfile(f.getCanonicalPath());
List<String> argList = Arrays.asList("--file","test.lua");
JsePlatform.luaMain(chunk, argList.toArray(new String[argList.size()]));
However i always get attempt to call nil where the code tries to access the arg table ( while i < table.getn(arg) do) - i tried other examples and they all result in the same error - luaj does not seem to set the "arg" table correctly - even a simply print arg[1] will not work.
LuaJ does not support table.getn anymore because it got removed in lua 5.1 - replace every occurances of table.getn with #varname - and init the args array with ocal args={...} at the top made it work.
Still, the code formatter does not really do what i expected it todo
There are two issues:
calls to table.getn(arg) should be replaced with #arg
the chunk's environment is not set up properly by luaj 3.0.1 so arg isn't set
However, as a workaround, you can capture the inputs using the varargs "..." syntax by adding a line at the top of codeformatter.lua such as
arg = {...}
Here is a code snippet to illustrate:
Globals globals = JsePlatform.standardGlobals();
LuaValue chunk = globals.load(
"arg = {...};" +
"print(#arg, arg[1], arg[2])");
JsePlatform.luaMain(chunk, new String[] {"--file","test.lua"});
Produces output:
2 --file test.lua

sanitizing a Lua table input

Let's say I want a Lua table that will be provided from a third party, not totally reliable, from a file or other IO source.
I get the table as a string, like "{['valid'] = 10}" and I can load it as
externalTable = loadstring("return " .. txtTable)()
But this opens a breach to code injection, ie.: txtTable = os.execute('rm -rf /')
So I did this sanitizing function:
function safeLoadTable(txtTable)
txtTable = tostring(txtTable)
if (string.find(txtTable, "(", 1, true))
then return nil end
local _start = string.find(txtTable, "{", 1, true)
local _end = string.find(string.reverse(txtTable), "}", 1, true)
if (_start == nil or _end == nil)
then return nil end
txtTable = string.sub(txtTable, _start, #txtTable - _end + 1)
print("cropped to ", txtTable)
local pFunc = loadstring("return " .. txtTable)
if (pFunc) then
local _, aTable = pcall(pFunc)
return aTable
end
end
In the worst case it should return nil.
Can this be considered safe against a "regular bad-intentioned person" :)
You could run the unsafe code in a sandbox.
Here is how a simple sandbox could look in Lua 5.1 (error handling omitted for brevity):
local script = [[os.execute("rm -rf /")]]
local env = { print=print, table=table, string=string }
local f, err = loadstring(script)
if err then
-- handle syntax error
end
setfenv(f, env)
local status, err = pcall(f)
if not status then
-- handle runtime error
end
In Lua 5.2 you can load the script into it's own environment using the load function.
The result would be a runtime error returned from pcall:
attempt to index global 'os' (a nil value)
EDIT
As Lorenzo Donati pointed out in the comments this is not a complete solution to stop rogue scripts. It essentially allows you to white-list functions and tables that are approved for user scripts.
For more info about handling rogue scripts I would suggest this SO question:
Embedded Lua - timing out rogue scripts (e.g. infinite loop) - an example anyone?
I don't think it is safe. Try this:
print(safeLoadTable [[{ foo = (function() print"yahoo" end)() } ]])
EDIT
or this, for more fun:
print(safeLoadTable [[{ foo = (function() print(os.getenv "PATH") end)() } ]])
I won't suggest the alternative of replacing that os.getenv with os.execute, though. :-)
The problem is not easy to solve. Code injection avoidance is not at all simple in this case because you are executing a piece of Lua code when doing that loadstring. No simple string matching technique is really safe. The only secure way would be to implement a parser for a subset of the Lua table syntax and use that parser on the string.
BTW, even Lua team stripped off the bytecode verifier from Lua 5.2 since they discovered that it was amenable to attacks, and bytecode is a far simpler language than Lua source code.
I created sandbox.lua for exactly this purpose. It'll handle both insecure stuff as well as DOS-type attacks, assuming that your environment has access to the debug facility.
https://github.com/kikito/sandbox.lua
Note that for now it is Lua 5.1-compatible only.
Running in sandbox isn't safe, inspecting source code is not very simple. An idea: inspect bytecode!
Emmm, actually that's not very simple either, but here is a lazy implementation: http://codepad.org/mGqQ0Y8q

How do I read an environment variable in Verilog/System Verilog?

How do I read an environment variable in Verilog ? (Running on a VCS simulator)
I am trying to accomplish
File=$fopen("$PATH/FileName","r");
$PATH is an environment variable.
You can simply use SystemVerilog DPI for getting environment.
And because getenv is a standard C library for every POSIX platform, so you do not need to implement your own getenv() equivalent function for the function definition again.
Example code in SV.
import "DPI-C" function string getenv(input string env_name);
module top;
initial begin
$write("env = %s\n", {getenv("HOME"), "/FileName"});
end
endmodule
Running
ncverilog -sv dpi.v
or
vcs -sverilog dpi.v
It will show
env = /home/user/FileName
And one more issue in your original question, PATH is a environment for executable search path and concatenate with ":" character. I think it should be an example here, not really "PATH" environment. Otherwise, your fopen file name could be "/bin:/usr/bin:/usr/local/bin/FileName", which is wrong.
You can use a simple PLI application to read an environment variable. Here's a sample, without any error checks:
#include <stdlib.h>
#include <string.h>
#include "vpi_user.h"
PLI_INT32 pli_getenv (PLI_BYTE8 * arg) {
vpiHandle tf_obj = vpi_handle (vpiSysTfCall, NULL);
vpiHandle arg_iter = vpi_iterate (vpiArgument, tf_obj);
vpiHandle arg1, arg2;
arg1 = vpi_scan (arg_iter);
arg2 = vpi_scan (arg_iter);
s_vpi_value vi, vo;
vi.format = vpiStringVal;
vpi_get_value (arg2, &vi);
vo.format = vpiStringVal;
vo.value.str = strdup (getenv (vi.value.str));
vpi_put_value (arg1, &vo, NULL, vpiNoDelay);
return 0;
}
The VCS documentation should explain how to link this into the simulator.
It is often simpler to use the Verilog preprocessor
File = $fopen(`PATH_FILENAME, "r");
Then invoke the simulator from your Makefile/shell script the specifying value to be substituted
$(SIM) -DPATH_FILENAME=\"$PATH/FileName\" blah.v ...
I use this with Icarus' iverilog often, vsim and friends probably support similar.
Quotes are escaped so that they are included in the substituted value, since the preprocessor will not substitute inside a literal value. For instance this combination does not work:
File = $fopen("`PATH_FILENAME", "r");
...
`$(SIM) -DPATH_FILENAME=$PATH/FileName blah.v ...`
Here I can see all answers, either they are using some DPI Or need some command line arguments. So I am sharing my answer with only SystemVerilog syntax. Answer is not specific to any simulator. But surely it is for Linux environment; for other OS we need to change $system commands.
We need to set this "logPath" system variable using some pre
processing script or by simulation script before we start our
simulation.
string myPath;
initial begin
//Writing System Variable To A File
$system("echo ${logPath} > logPath.txt");
//Opening that file and reading to a string variable
fh = $fopen ("./logPath.txt", "r");
void'($fscanf(fh,"%s",myPath));
//Appending File Name To That Path
myPath = {myPath,"/note.txt"};
//Closed and remove this temporary file
$fclose(fh);
$system("rm -rf logPath.txt");
//Open a file at the path that you have extracted from System Variable
//Do whatever you want now
fh = $fopen (myPath, "w");
repeat(10) begin
$fdisplay (fh, "%t %M: Write Line Number =|%0d| ", $time, i);
i++;
end
$fclose(fh);
end

Print list of ALL environment variables

I would like to print a list of all environment variables and their values. I searched the Stackoverflow and the following questions come close but don't answer me:
How to discover what is available in lua environment? (it's about Lua environment not the system environment variables)
Print all local variables accessible to the current scope in Lua (again about _G not the os environment variables)
http://www.lua.org/manual/5.1/manual.html#pdf-os.getenv (this is a good function but I have to know the name of the environment variable in order to call it)
Unlike C, Lua doesn't have envp** parameter that's passed to main() so I couldn't find a way to get a list of all environment variables. Does anybody know how I can get the list of the name and value of all environment variables?
Standard Lua functions are based on C-standard functions, and there is no C-standard function to get all the environment variables. Therefore, there is no Lua standard function to do it either.
You will have to use a module like luaex, which provides this functionality.
This code was extracted from an old POSIX binding.
static int Pgetenv(lua_State *L) /** getenv([name]) */
{
if (lua_isnone(L, 1))
{
extern char **environ;
char **e;
if (*environ==NULL) lua_pushnil(L); else lua_newtable(L);
for (e=environ; *e!=NULL; e++)
{
char *s=*e;
char *eq=strchr(s, '=');
if (eq==NULL) /* will this ever happen? */
{
lua_pushstring(L,s);
lua_pushboolean(L,0);
}
else
{
lua_pushlstring(L,s,eq-s);
lua_pushstring(L,eq+1);
}
lua_settable(L,-3);
}
}
else
lua_pushstring(L, getenv(luaL_checkstring(L, 1)));
return 1;
}
You can install the lua-posix module. Alternatively, RedHat installations have POSIX routines built-in, but to enable them, you have to do a trick:
cd /usr/lib64/lua/5.1/
# (replace 5.1 with your version)
ln -s ../../librpmio.so.1 posix.so
# (replace the "1" as needed)
lua -lposix
> for i, s in pairs(posix.getenv()) do print(i,s,"\n") end
The trick is in creating a soft-link to the RPM's "io" directory and to naming the soft-link the same name of the library LUA will attempt to open. If you don't do this, you get:
./librpmio.so: undefined symbol: luaopen_librpmio
or similar.
local osEnv = {}
for line in io.popen("set"):lines() do
envName = line:match("^[^=]+")
osEnv[envName] = os.getenv(envName)
end
this would not work in some cases, like "no valid shell for the user running your app"
An easy 2 liner:
buf = io.popen("env", '*r')
output = buf:read('*a')
print(output) -- or do whatever

lua how require works

I'm using a graphics library that lets you program in Lua. I have a need for the A* pathfinding library so I found one online. It's just 1 lua file that does the pathfinding and 1 example file. In the example file it uses the object like:
-- Loading the library
local Astar = require 'Astar'
Astar(map,1) -- Inits the library, sets the OBST_VALUE to 1
I run the script and everything works. So now I add the Astar.lua file to the path location where my graphics engine is running and do the same thing and I get the error on the Astar(map, 1) line:
"attempt to call local 'AStar' (a number value)
Any ideas why I would be getting that error when I'm doing the same thing as the example that comes with this AStar lib?
Here is a little of the AStar file
-- The Astar class
local Astar = {}
setmetatable(Astar, {__call = function(self,...) return self:init(...) end})
Astar.__index = Astar
-- Loads the map, sets the unwalkable value, inits pathfinding
function Astar:init(map,obstvalue)
self.map = map
self.OBST_VALUE = obstvalue or 1
self.cList = {}
self.oList = {}
self.initialNode = false
self.finalNode = false
self.currentNode = false
self.path = {}
self.mapSizeX = #self.map[1]
self.mapSizeY = #self.map
end
So note that when I run this from my graphics engine it's returning 1, but when run from the example that it came with it's returning a table, which is what it should be returning. So not sure why it would only be returning 1.
How is Astar getting added to the package.loaded table for the example script, as opposed to your code?
QUICK LUA SYNTACTIC SUGAR REVIEW:
func 'string' is equivalent to func('string')
tabl.ident is equivalent to tabl['ident']
When you run a script using require('Astar'), this is what it does:
checks if package.loaded['Astar'] is a non-nil value.
If it is, it returns this value. Otherwise it continues down this list.
Runs through filenames of the patterns listed in package.path (and package.cpath), with '?' replaced with 'Astar', until it finds the first file matching the pattern.
Sets package.loaded['Astar'] to true.
Runs the module script (found via path search above- for the sake of this example we'll assume it's not a C module) with 'Astar' as an argument (accessible as ... in the module script).
If the script returns a value, this value is placed into package.loaded['Astar'].
The contents of package.loaded['Astar'] are returned.
Note that the script can load the package into package.loaded['Astar'] as part of its execution and return nothing.
As somebody noted in the comments above, your problem may come from loading the module using 'AStar' instead of 'Astar'. It's possible that Lua is loading this script using this string (since, on the case-insensitive Windows, a search for a file named "AStar.lua" will open a file called "Astar.lua"), but the script isn't operating with that (by using a hard-coded "Astar" instead of the "AStar" Lua is loading the script under).
You need to add return Astar at the end of Astar.lua.

Resources