how to make a proc available compile time and run time - compile-time

I have a nim proc that dedents a multiline string based on the whitespace
of the first indented line:
import strutils
proc dedent(s: string): string
{.noSideEffect.} =
var prefix = ""
for line in s.splitLines(keepEol=true):
if prefix == "" and line.len > 0:
for j in 0..<line.len:
if line[j] != ' ':
prefix = line[0..j-1]
break
if line.startsWith(prefix):
result.add(line[prefix.len .. high(line)])
else:
result.add(line)
when isMainModule:
echo dedent """
cat:
- meow
- purr
dog:
- bark
- drool
"""
it nicely outputs:
cat:
- meow
- purr
dog:
- bark
- drool
but inspecting the intermediate C code, I see:
STRING_LITERAL(TM_9amIjLnWbK7OR9aPA8dicbaQ_14, " cat:\012 - meow\012 - purr\012 \012 "
" dog:\012 - bark\012 - drool\012 ", 112);
so the dedenting is done at run-time. I can add the compileTime pragma to the proc:
proc dedent(s: string): string
{.noSideEffect,compileTime.} =
and then the C output changes to:
STRING_LITERAL(TM_9amIjLnWbK7OR9aPA8dicbaQ_3, "cat:\012 - meow\012 - purr\012\012dog:\012 - bark\012 - drool\012", 48);
Which is exactly what I want, multi-line strings that are indented to their surrounding on every line, but don't have that extra indentation in the executable.
But adding that pragma, I can no longer access dedent at run-time, e.g. when adding:
import os
if paramCount() > 0:
for i in 1..paramCount():
echo dedent paramStr(i)
to isMainModule, you get the error:
Error: request to generate code for .compileTime proc: dedent
I looked at the source for splitLines in strutils.nim to see if there was some other pragma that I could apply, but I did not find anything that would work.
I now about the static statement, but would prefer that the compiler optimises this at compile time without me having to sprinkle this in.
How can I get this to work both compile-time as well as run-time without reverting to the use of static?
Do I need to compile the proc from a seperate .nim module? Or is there a compiler option, pragma, something else, that I am missing?

You need to pass the --implicitStatic:on flag to the Nim compiler to enable implicit compile time evaluation of procedures.
These procedures cannot have the compileTime pragma.

To force the evaluate of any expression at compile-time, you can use static in the following way:
when isMainModule:
echo static(dedent"""
cat:
- meow
- purr
dog:
- bark
- drool
""")
All expressions assigned to constants are also evaluated at compile-time, so this is another way to achieve the same.

Related

Passing strings to .wasm module

I've been stuck on this for a while now and I cannot seem to find good resources to my problem. I am coming from and "only C" background, so most of the web dev stuff is completely new for me.
I wrote a C function float editDistance(char *str1, char *str2) that returns the edit distance of 2 char arrays. Right now the goal is to successfully call this function from a JS environment.
After ensuring that the code works with the recommended Emscipten ccall method, I decided to move on. Now
I use Emscripten to compile the C code with flags -O3, -s WASM=1, -s EXPORTED_FUNCTIONS="['_editDistance']", and -s SIDE_MODULE=1 -s to Wasm. The JS code I'm trying to wrap around my WebAssembly is:
// Allocate memory for the wasm module to run in. (65536*256 bit)
let wasmMemory = new WebAssembly.Memory({
initial: 256
});
let info = {
env: {
abort: function() {},
memoryBase: 0,
tableBase: 0,
memory: wasmMemory,
table: new WebAssembly.Table({initial: 2, element: 'anyfunc'}),
}
}
// Define the strings
let str1 = "abcd";
let str2 = "abcd";
// Allocate memory on the wasm partition for the HEAPU8
let HEAPU8 = new Uint8Array(wasmMemory.buffer);
// Create the char arrays on the heap from the strings
let stackPtr = 0;
let str1Ptr = stackPtr;
stackPtr = stringToASCIIArray(str1, HEAPU8, stackPtr);
let str2Ptr = stackPtr;
stackPtr = stringToASCIIArray(str2, HEAPU8, stackPtr);
// Read the wasm file and instantiate it with the above environment setup. Then
// call the exported function with the string pointers.
let wasmBinaryFile = 'bin/edit_distanceW.wasm';
fetch(wasmBinaryFile, {credentials:"same-origin"})
.then((response) => response.arrayBuffer())
.then((binary) => WebAssembly.instantiate(binary,info))
.then((wa) => alert(wa.instance.exports._editDistance(str1Ptr, str2Ptr)));
// Converts a string to an ASCII byte array on the specified memory
function stringToASCIIArray(str, outU8Array, idx){
let length = str.length + 1;
let i;
for(i=0; i<length; i++){
outU8Array[idx+i] = str.charCodeAt(i);
}
outU8Array[idx+i]=0;
return (idx + length);
}
The generated wasm file when converted to wat demands these imports:
(import "env" "abort" (func (;0;) (type 0)))
(import "env" "memoryBase" (global (;0;) i32))
(import "env" "tableBase" (global (;1;) i32))
(import "env" "memory" (memory (;0;) 256))
(import "env" "table" (table (;0;) 2 anyfunc))
.. and exports these:
(export "__post_instantiate" (func 7))
(export "_editDistance" (func 9))
(export "runPostSets" (func 6))
(elem (;0;) (get_global 1) 8 1))
Now, when I test the code the strings are passed to the C module without a problem. A few function calls are even made on them (strLen) before things go south. In the C function there is this nasty nested loop that does the main computation, iterating thru a 2D array while reading the characters from the strings (C code just been ported from a paper with an ugly pseudo code, so pardon me the variable names):
do{
for(p=0; p<editDistance; p++){
// Do stuff
}
// Do more stuff
editDistance++;
} while(fkp[len2*2-len1][editDistance] != len1);
Before the function enters the for() loop, the module still has the strings on memory str1Ptr=0x00 and str2Ptr=0x05 with the correct length and content. On the contrary, immediately after entering the for() loop the memory gets overwritten by garbage (mostly 0s), corrupting the end result. I suspect some stack saving and restoration problems on the scope change, as the exact same code compiled to my PC using gcc works like a charm.
Any idea what setup I'm missing that hinders the correct completion of the C function?
If you are starting out you probably want to use the emscripten-generated JS glue. That is, don't use SIDE_MODULE=1 and instead output to a files calle .js. The emscripten compiler will then generate both a .js and a .wasm file. You can then include the .js file in your project and it will handle all the loading and setup for you.
If you try to load the wasm file yourself, you will need to do a lot of work to replicate the emscripten environment, which will require a lot of internal details of emscripten. Also, those internal details of subject to change when you update to the new version of emscripten so you are creating more work for yourself.

LuaJ does not supply command line arguments correctly

I tried the utility method provided by luaj to call a lua file with command line args (this one http://lua-users.org/wiki/SourceCodeFormatter)
Globals globals = JsePlatform.standardGlobals();
String script ="src/codeformatter.lua";
File f = new File(script);
LuaValue chunk = globals.loadfile(f.getCanonicalPath());
List<String> argList = Arrays.asList("--file","test.lua");
JsePlatform.luaMain(chunk, argList.toArray(new String[argList.size()]));
However i always get attempt to call nil where the code tries to access the arg table ( while i < table.getn(arg) do) - i tried other examples and they all result in the same error - luaj does not seem to set the "arg" table correctly - even a simply print arg[1] will not work.
LuaJ does not support table.getn anymore because it got removed in lua 5.1 - replace every occurances of table.getn with #varname - and init the args array with ocal args={...} at the top made it work.
Still, the code formatter does not really do what i expected it todo
There are two issues:
calls to table.getn(arg) should be replaced with #arg
the chunk's environment is not set up properly by luaj 3.0.1 so arg isn't set
However, as a workaround, you can capture the inputs using the varargs "..." syntax by adding a line at the top of codeformatter.lua such as
arg = {...}
Here is a code snippet to illustrate:
Globals globals = JsePlatform.standardGlobals();
LuaValue chunk = globals.load(
"arg = {...};" +
"print(#arg, arg[1], arg[2])");
JsePlatform.luaMain(chunk, new String[] {"--file","test.lua"});
Produces output:
2 --file test.lua

How to get the output of python script executed from a ruby method

I am trying to run a python script from ruby method. I am running this method as a rake task within a Rails app. I am using the solution mentioned here:
def create
path = File.expand_path('../../../../GetOrders', __FILE__)
output = `"python2 " + path + "/parse.py"`
print output
str = JSON.parse(output)
print str
end
EDIT: This works:
output = `python2 #{path}/parse.py`
EDIT2:
Using the python script i am trying to pass a list of dictionaries to the ruby function. The python script looks something like:
import xml.etree.ElementTree as ET
import json
def parse():
tree = ET.parse('response.xml')
root = tree.getroot()
namespaces = {'resp': 'urn:ebay:apis:eBLBaseComponents'}
order_array = root.find("resp:OrderArray", namespaces=namespaces)
detailsList = []
for condition:
details["key1"] = value1
details["key2"] = value2
detailsList.append(details)
output = json.dumps(detailsList)
return output
print parse()
Could someone explain what i am doing wrong and how can I fix this. Thanks
When you do this:
output = `python2 #{path}/parse.py`
output will be assigned the standard output of the python script, but that script isn't writing anything to standard output; the json data that's the return value of the parse() call is simply discarded. You seem to be expecting the execution of the script to have a "return value" that's the return value of the script's last expression, but that's not how processes work.
You probably want to replace the parse() call at the end of the script with print parse().
You are calling this exact line on the shell:
"python2 -path- /parse.py"
which the shell interprets as a single command: python2 (with a space at the end).
Try using string interpolation, which works with the backtick operator:
output = `python2 #{path}/parse.py`
Imagine typing this exact string:
"python2 " + path + "/parse.py"
into your shell (e.g. bash). It would look for a program named "python2 " and give it four arguments
+
path
+
/parse.y
You can't put arbitrary Ruby code inside a backtick string the same way you can't put arbitrary code in normals strings. You must use string interpolation.

sanitizing a Lua table input

Let's say I want a Lua table that will be provided from a third party, not totally reliable, from a file or other IO source.
I get the table as a string, like "{['valid'] = 10}" and I can load it as
externalTable = loadstring("return " .. txtTable)()
But this opens a breach to code injection, ie.: txtTable = os.execute('rm -rf /')
So I did this sanitizing function:
function safeLoadTable(txtTable)
txtTable = tostring(txtTable)
if (string.find(txtTable, "(", 1, true))
then return nil end
local _start = string.find(txtTable, "{", 1, true)
local _end = string.find(string.reverse(txtTable), "}", 1, true)
if (_start == nil or _end == nil)
then return nil end
txtTable = string.sub(txtTable, _start, #txtTable - _end + 1)
print("cropped to ", txtTable)
local pFunc = loadstring("return " .. txtTable)
if (pFunc) then
local _, aTable = pcall(pFunc)
return aTable
end
end
In the worst case it should return nil.
Can this be considered safe against a "regular bad-intentioned person" :)
You could run the unsafe code in a sandbox.
Here is how a simple sandbox could look in Lua 5.1 (error handling omitted for brevity):
local script = [[os.execute("rm -rf /")]]
local env = { print=print, table=table, string=string }
local f, err = loadstring(script)
if err then
-- handle syntax error
end
setfenv(f, env)
local status, err = pcall(f)
if not status then
-- handle runtime error
end
In Lua 5.2 you can load the script into it's own environment using the load function.
The result would be a runtime error returned from pcall:
attempt to index global 'os' (a nil value)
EDIT
As Lorenzo Donati pointed out in the comments this is not a complete solution to stop rogue scripts. It essentially allows you to white-list functions and tables that are approved for user scripts.
For more info about handling rogue scripts I would suggest this SO question:
Embedded Lua - timing out rogue scripts (e.g. infinite loop) - an example anyone?
I don't think it is safe. Try this:
print(safeLoadTable [[{ foo = (function() print"yahoo" end)() } ]])
EDIT
or this, for more fun:
print(safeLoadTable [[{ foo = (function() print(os.getenv "PATH") end)() } ]])
I won't suggest the alternative of replacing that os.getenv with os.execute, though. :-)
The problem is not easy to solve. Code injection avoidance is not at all simple in this case because you are executing a piece of Lua code when doing that loadstring. No simple string matching technique is really safe. The only secure way would be to implement a parser for a subset of the Lua table syntax and use that parser on the string.
BTW, even Lua team stripped off the bytecode verifier from Lua 5.2 since they discovered that it was amenable to attacks, and bytecode is a far simpler language than Lua source code.
I created sandbox.lua for exactly this purpose. It'll handle both insecure stuff as well as DOS-type attacks, assuming that your environment has access to the debug facility.
https://github.com/kikito/sandbox.lua
Note that for now it is Lua 5.1-compatible only.
Running in sandbox isn't safe, inspecting source code is not very simple. An idea: inspect bytecode!
Emmm, actually that's not very simple either, but here is a lazy implementation: http://codepad.org/mGqQ0Y8q

How do I read an environment variable in Verilog/System Verilog?

How do I read an environment variable in Verilog ? (Running on a VCS simulator)
I am trying to accomplish
File=$fopen("$PATH/FileName","r");
$PATH is an environment variable.
You can simply use SystemVerilog DPI for getting environment.
And because getenv is a standard C library for every POSIX platform, so you do not need to implement your own getenv() equivalent function for the function definition again.
Example code in SV.
import "DPI-C" function string getenv(input string env_name);
module top;
initial begin
$write("env = %s\n", {getenv("HOME"), "/FileName"});
end
endmodule
Running
ncverilog -sv dpi.v
or
vcs -sverilog dpi.v
It will show
env = /home/user/FileName
And one more issue in your original question, PATH is a environment for executable search path and concatenate with ":" character. I think it should be an example here, not really "PATH" environment. Otherwise, your fopen file name could be "/bin:/usr/bin:/usr/local/bin/FileName", which is wrong.
You can use a simple PLI application to read an environment variable. Here's a sample, without any error checks:
#include <stdlib.h>
#include <string.h>
#include "vpi_user.h"
PLI_INT32 pli_getenv (PLI_BYTE8 * arg) {
vpiHandle tf_obj = vpi_handle (vpiSysTfCall, NULL);
vpiHandle arg_iter = vpi_iterate (vpiArgument, tf_obj);
vpiHandle arg1, arg2;
arg1 = vpi_scan (arg_iter);
arg2 = vpi_scan (arg_iter);
s_vpi_value vi, vo;
vi.format = vpiStringVal;
vpi_get_value (arg2, &vi);
vo.format = vpiStringVal;
vo.value.str = strdup (getenv (vi.value.str));
vpi_put_value (arg1, &vo, NULL, vpiNoDelay);
return 0;
}
The VCS documentation should explain how to link this into the simulator.
It is often simpler to use the Verilog preprocessor
File = $fopen(`PATH_FILENAME, "r");
Then invoke the simulator from your Makefile/shell script the specifying value to be substituted
$(SIM) -DPATH_FILENAME=\"$PATH/FileName\" blah.v ...
I use this with Icarus' iverilog often, vsim and friends probably support similar.
Quotes are escaped so that they are included in the substituted value, since the preprocessor will not substitute inside a literal value. For instance this combination does not work:
File = $fopen("`PATH_FILENAME", "r");
...
`$(SIM) -DPATH_FILENAME=$PATH/FileName blah.v ...`
Here I can see all answers, either they are using some DPI Or need some command line arguments. So I am sharing my answer with only SystemVerilog syntax. Answer is not specific to any simulator. But surely it is for Linux environment; for other OS we need to change $system commands.
We need to set this "logPath" system variable using some pre
processing script or by simulation script before we start our
simulation.
string myPath;
initial begin
//Writing System Variable To A File
$system("echo ${logPath} > logPath.txt");
//Opening that file and reading to a string variable
fh = $fopen ("./logPath.txt", "r");
void'($fscanf(fh,"%s",myPath));
//Appending File Name To That Path
myPath = {myPath,"/note.txt"};
//Closed and remove this temporary file
$fclose(fh);
$system("rm -rf logPath.txt");
//Open a file at the path that you have extracted from System Variable
//Do whatever you want now
fh = $fopen (myPath, "w");
repeat(10) begin
$fdisplay (fh, "%t %M: Write Line Number =|%0d| ", $time, i);
i++;
end
$fclose(fh);
end

Resources