I'm trying to serialize and deserialize a Lua closure
my basic understanding is that the below factory should generate closures (and that Lua doesn't much distinguish between functions and closures -- i.e. there is no type 'closure')
> function ffactory(x) return function() return x end end
> f1 = ffactory(5)
> print(f1())
5 <-- so far so good
> s = string.dump(f1)
> f2 = load(s)
> print(f2())
table: 00000000002F7BA0 <-- expected the integer 5
> print(f2()==_ENV)
true <-- definitely didn't expect this!
I expected the integer 5 to be serialized with f1. Or, if string.dump can't handle closures, i expected an error.
I get quite different (but more what I expected) results with a mild change. It looks like f2 is indeed a closure, but string.dump didn't attempt to serialize the value of x at the time it was serialized.
The docs don't help me much. (what do they mean by "...with new upvalues"?)
> function ffactory(x) return function() return x+1 end end
> f1 = ffactory(5)
> print(f1())
6 <-- good
> s = string.dump(f1)
> f2 = load(s)
> print(f2())
stdin:1: attempt to perform arithmetic on upvalue 'x' (a table value)
stack traceback:
stdin:1: in function 'f2'
stdin:1: in main chunk
[C]: in ?
You can do something like this to save/restore those upvalues (note it doesn't handle upvalues shared between different functions):
local function capture(func)
local vars = {}
local i = 1
while true do
local name, value = debug.getupvalue(func, i)
if not name then break end
vars[i] = value
i = i + 1
end
return vars
end
local function restore(func, vars)
for i, value in ipairs(vars) do
debug.setupvalue(func, i, value)
end
end
function ffactory(x) return function() return x end end
local f1 = ffactory(5)
local f2 = (loadstring or load)(string.dump(f1))
restore(f2, capture(f1)) --<-- this restored upvalues from f1 for f2
print(f1(), f2())
This works under both Lua 5.1 and Lua 5.2.
Note an interesting result if you change ffactory slightly (added math.abs(0); anything that uses global table in any way will do):
function ffactory(x) return function() math.abs(0) return x end end
Now if you restore upvalues you get the same result, but if you don't restore upvalues you get a run-time error under Lua 5.2:
lua.exe: upvalues.lua:19: attempt to index upvalue '_ENV' (a nil value)
stack traceback:
upvalues.lua:19: in function 'f2'
upvalues.lua:24: in main chunk
[C]: in ?
the docs are pretty clear. string.dump doesn't handle closures using upvalues. this is because the upvalues could be anything (including userdata, and how would Lua know how to serialize that?)
upvalues are the external variables that are local to a function due to scoping/closures. since x in your example is an upvalue to the function returned by ffactory, it does not get serialized.
if you want to support this somehow, you would have to store the upvalues yourself and set them again after you've deserialized the function, like this:
function ffactory(x)
return function() return x+1 end
end
local f1 = ffactory(5)
print(f1())
local s = string.dump(f1)
f2 = loadstring(s)
debug.setupvalue(f2, 1, 5)
print(f2())
Related
This question has some reference to the question Evaluating expression in Lua in Mathematics Environment
The following code works.
tbl = {}
tbl.sin = math.sin
tbl.cos = math.cos
function mathEval(exp)
return load("return " .. exp, exp, "t", tbl)()
end
print(mathEval("sin(0)"))
print(mathEval("sin(0)+cos(1)+2^2"))
However, the following code does not work.
tbl = {}
tbl.sin = math.sin
tbl.cos = math.cos
function mathEval(exp)
return load("return " .. tostring(exp), tostring(exp), "t", tbl)()
end
print(mathEval(sin(0)))
print(mathEval(sin(0)+cos(1)+2^2))
I want to evaluate expressions without using quotes. How can that be done?
The problem with the line print(mathEval(sin(0)+cos(1)+2^2)) is that the argument of mathEval is evaluated before mathEval runs, so evaluating the variables sin and cos can not be deferred to the environment of mathEval; that is, mathEval gets a value, and no expression to evaluate at all!
First of all, one option to evaluate such mathematical expressions without the use of mathEval would be to simply temporarily change your environment:
local prev_env = _ENV -- this is needed to restore the environment later on
_ENV = tbl -- enter custom environment
local result = sin(0)+cos(1)+2^2
_ENV = prev_env -- restore environment
print(result)
if you want mathEval as a convenience helper, you'll have to pass the expression as a function returning the value to the expression such that calling the function will evaluate the expression; this allows you to defer the initialization. You'll have to use a powerful function called setfenv which allows you to change the environment of func; this was unfortunately removed in favor of _ENV in Lua 5.2 and later. The code then becomes trivial:
local function mathEval(func)
setfenv(func, tbl)
return func
end
mathEval(function() return sin(0)+cos(1)+2^2 end)
setfenv can be replicated in Lua 5.2 using the debug library, since Lua internally implements _ENV as an upvalue, as shown by Leafo:
local function setfenv(fn, env)
local i = 1
while true do
local name = debug.getupvalue(fn, i)
if name == "_ENV" then
debug.upvaluejoin(fn, i, (function()
return env
end), 1)
break
elseif not name then
break
end
i = i + 1
end
return fn
end
I assume you do not want to evaluate the expression before passing the result as an argument? Then you could wrap your expression into a function, which is then lazily called. It replaces the environment with tbl, executes the function, and reverts the environment.
tbl = {}
tbl.sin = math.sin
tbl.cos = math.cos
function mathEval(func)
local old = _ENV
_ENV = tbl
local r = func()
_ENV = old
return r
end
print(mathEval(function() return sin(0)+cos(1)+2^2 end))
You are passing sin(0) and sin(0)+cos(1)+2^2 to the mathEval(exp) function, but since sin and cos are not global variables and the math library is not being passed as an environment, Lua is unable to find them.
If you want to use math functions with the mathEval function, you can use the tbl table created before, to call the math functions, for example:
print(mathEval("tbl.sin(0)"))
print(mathEval("tbl.sin(0)+tbl.cos(1)+2^2"))
or alternatively you can pass the math library or the math functions as an upvalue to the load() function:
function mathEval(exp)
return load("return " .. tostring(exp), tostring(exp), "t", _ENV)()
end
print(mathEval("math.sin(0)"))
print(mathEval("math.sin(0)+math.cos(1)+2^2"))
It is possible to do this without quotes, this should work:
expression = string.format("%s(%d)", "math.sin", 0)
fn = load( "return " .. expression)
result = fn()
print(result) -- Output: 0
In this example, the string.format() function is used to create the string "math.sin(0)". The resulting string is then passed to the load() function as before.
In this way you don't need to use quotes on the string, and you can use placeholders to concatenate the variables.
I'm struggling to understand why is the __gc metamethod is not being called here. I immediately removed the reference to x and triggered a GC, so why?
local obj = {}
obj.__index = obj
function obj:__gc()
print("gc")
end
function new()
return setmetatable({}, obj)
end
local x = new()
x = nil
collectgarbage()
I expect that the GC would collect that x and so it would trigger the __gc metamethod associated with it.
This is just a minimal reproducible example but to make my goals clear I want to wrap a C data in LuaJIT's ffi to deallocate its memory upon garbage collection.
I just found about ffi.metatype which allows me to attach a metatable with __gc e.g:
local ffi = require("ffi")
local mt = {}
local obj = ffi.metatype("struct { int x; }", mt)
function mt:__gc()
print("gc")
end
function new()
return obj(5)
end
local x = new()
x = nil
collectgarbage()
Since I need to wrap a C pointer, this is just right for me.
I don't think __gc is supported in the current version of LuaJIT, but it does seem to be planned as part of this really old proposal for a new GC:
https://github.com/LuaJIT/LuaJIT/issues/38, which is linked to this issue: https://github.com/LuaJIT/LuaJIT/issues/47#issuecomment-132403863
I want to optimize a Fibonacci function but using table index, and memoization seems a good method (iteration is as good)... However, soon I ran into a problem: I can't decide whether a key is in a table. How can I do that?
local fib = {}
function fib_index(t, k)
if k == 0 or k == 1 then
t[k] = k
else
t[k] = t[k-1] + t[k-2]
end
return t[k]
end
setmetatable(fib, {__index = fib_index})
Your code will run just fine in it's current state. The reason behind that is that already strored values have highter priority than __index metamethod, so if value does exist, it's returned immideately. There are few optimisations which can make your code look better:
local fib = setmetatable({1, 1}, {__index = function(t,k)
assert(k > 0, 'Invalid fib index') -- Most basic check
local res = t[k-1] + t[k-2]
t[k] = res
return res
end})
Here I remove function declaration at all (if you want to reuse it, consider making your function local with local function instead of function) and made code easier by adding inital values directly into table declaration (no index 0 to keep it lua way, also no zero in results) and utilizing the fact that setmetatable return originally passed table. You can remove assert if you want to, but it's probably good idea to see meaningful error message instead of "stack overflow".
And if you really wanna check does value exist in table (this code does not require this), use rawget:
rawget(fib, 10) == nil
will tell you is 10 already calculated and cached.
I am trying to use the redis-lua library within copas. It requires some patching.
One problem is that redis-lua defines some iterators as coroutines, but these iterators perform network operations that can yield.
So, coroutine.yield is used for two very different things: for the iterator, and for copas. As network calls are nested within iterators, network yields are intercepted by the coroutine.wrap of the iterator, instead of being intercepted by copas.
The example below shows the problem :
local function iterator ()
for i = 1, 2 do
if i == 2 then coroutine.yield () end -- network yield
coroutine.yield () -- iterator yield
end
end
local citerator = coroutine.wrap (iterator)
local function loop () -- use of the iterator within a copas thread
while citerator () do end
end
local cloop = coroutine.create (loop)
while coroutine.resume (cloop) do end -- same as copas loop, executes the cloop thread
Is there a "standard" solution to this problem, still allowing to use coroutines for iterators?
I was able to make a small example work by "tagging" the yields (see below), but it is incompatible with existing code. I can leave the copas code unmodified, but have to update iterators in redis-lua.
local function wrap (f, my_tag)
-- same as coroutine.wrap, but uses my_tag to yield again
local co = coroutine.create (f)
return function ()
local t = table.pack (coroutine.resume (co))
local code = t [1]
local tag = t [2]
table.remove (t, 1)
table.remove (t, 1)
if tag == nil then
return
elseif my_tag == tag then
return table.unpack (t)
else
coroutine.yield (tag, table.unpack (t))
end
end
end
local Iterator = {} -- tag for iterator yields
local Network = {} -- tag for network yields
local function iterator ()
for i = 1, 2 do
if i == 2 then coroutine.yield (Network, i) end
coroutine.yield (Iterator, i)
end
end
local citerator = wrap (iterator, Iterator)
local function loop ()
while citerator () do end
end
local cloop = wrap (loop, Network)
while cloop () do end
Is there a better solution?
Lua coroutine always yield to the last thread they were resumed from. The Copas socket functions expect to yield back to the Copas event loop, but instead they get stuck with the coroutine used to implement the redis-lua iterators. Unfortunately there isn't much you can do to fix that except changing the code of the redis-lua iterators. The reason why no one has done so yet is that until Lua 5.2 (LuaJIT can do it as well) it wasn't even possible to yield from an iterator function (the iterator yields in redis-lua work fine because they never leave the iterator function, but you cannot yield beyond the for loop like the Copas socket functions would try to do).
Your idea about using a tag value for distinguishing the iterator yields from the rest is good. You just have to make sure that you pass all yields not intended for the iterator function to the coroutine one level up, including any arguments/return values of coroutine.yield and coroutine.resume (the later is implicit when calling the coroutine.wraped function).
More specifically, if you have code like this in redis-lua:
-- ...
return coroutine.wrap( function()
-- ...
while true do
-- ...
coroutine.yield( some_values )
end
end )
You change it to:
-- ...
local co_func = coroutine.wrap( function()
-- ...
while true do
-- ...
coroutine.yield( ITERATOR_TAG, some_values ) -- mark all iterator yields
end
return ITERATOR_TAG -- returns are also intended for the iterator
end )
return function()
return pass_yields( co_func, co_func() ) -- initial resume of the iterator
end
The ITERATOR_TAG and the pass_yields function go somewhere near the top of redis.lua:
local ITERATOR_TAG = {} -- unique value to mark yields/returns
local function pass_yields( co_func, ... )
if ... == ITERATOR_TAG then -- yield (or return) intended for iterator?
return select( 2, ... ) -- strip the ITERATOR_TAG from results and return
else
-- pass other yields/resumes back and forth until we hit another iterator
-- yield (or return); using tail recursion here instead of a loop makes
-- handling vararg lists easier.
return pass_yields( co_func, co_func( coroutine.yield( ... ) ) )
end
end
AFAIK, the redis-lua developers are planning to tag another release by the end of the year, so they probably will be grateful for pull requests.
In your first example, you are using wrap(loop). I suppose this is copas' wrap, because there is no reference to copas in this code...
However, you are supposed to copas.wrap() a socket, but your loop is a function!
See copas documentation for a good introduction.
I am creating functions (of x) from a string in Lua. The code I am using is
function fcreate(fs)
return assert(loadstring("return function (x) return " .. fs.." end"))()
end
This works for globals, e.g.
u=fcreate("math.sin(x)")
does the right thing.
However, it does not seem to like local variables. So
local c=1
u=fcreate("math.sin(x)+c")
will not work because c is local.
Is this fixable?
"loadstring does not compile with lexical scoping", so no, it can't see locals outside the loadstring call.
Is this fixable?
That depends. Why are you using loadstring in the first place? Lua supports closures as first class values, so I can't see from your example why you'd need loadstring.
Your example:
u = fcreate("math.sin(x)+c")
Can be rewritten without the need for loadstring or your fcreate function:
u = function(x) return math.sin(x)+c end
Which of course is the same as:
function u(x) return math.sin(x) + c end
I can see a case for loadstring if you have user-configurable expressions that you wanted to compile into some other function, but your case with the local c suggests that's not the case. Are you trying to make some kinda of home-rolled lamda syntax?
Can't be done in any reasonable way. For an example of why, look at this:
function makefunction(name)
local a = 1
local b = 2
local c = 3
-- ...
return assert(loadstring("return " .. name))
end
local a = 4
local func = makefunction("a")
print(func())
If this worked, what is printed? 1 or 4? Does it capture the variable from the place where the function was loaded, even though that function doesn't exist anymore? Or does it look it up from the place where it was called?
The first would mean that the function is lexically scoped wherever it's created. Being able to access the variable after the function has exited means that the variable would need to be promoted into an upvalue dynamically, which is not something that Lua can do at the moment. As it is now, Lua can see every access to a local variable during compilation, so it knows which variables to turn into upvalues (at a performance hit) and which to keep as locals.
The second would mean that variable accesses inside a loadstring'd function would work completely different than every other access in Lua: Lua uses lexical scoping, not dynamic scoping. It'd be a huge implementation change in Lua, and an extremely inconsistent one.
So, neither is supported. You can control the environment of a dynamically loaded function, using setfenv in Lua 5.1 or the env parameter of load(...) in Lua 5.2, but neither of those let you access local variables automatically.
Something you could do if you don't need to mutate the local variables is to pass those values as arguments to the generated function. You would still need to manually specify the variables to close over but its better then nothing.
For example, you can build up your closure to look like
return (function(a,b,c)
return function(x) return print(a, x) end
end)(...)
We might do that by changing your function to look like
function fcreate(variables, fs)
local varnames = {}
local varvalues = {}
local nvars = 0
for n,v in pairs(variables) do
nvars = nvars + 1
table.insert(varnames, n)
table.insert(varvalues, v)
end
local chunk_str = (
'return (function(' .. table.concat(varnames, ',') .. ') ' ..
'return function(x) return ' .. fs .. ' end ' ..
'end)(...)'
)
return assert( loadstring(chunk_str) )( unpack(varvalues, 1, nvars) )
end
local a = 1;
local f = fcreate({a=a}, 'x+a')
print(f(1), f(2))