I have a ARM CPU (SAM3S) running Lua. I have several static Librarys written in Lua. Now I run out of RAM. It is possible to move and run the Lua opcodes from RAM to FLASH?
Thank you
AR
I've written a on-demand compilation wrapper some time ago. It's hilariously easy and can be adapted to your use-case.
lib = setmetatable({}, {
__mode = "v", -- important (!) allows the Lua GC to collect unused entries
__index = function(self, k)
-- put code here to load from FLASH
-- in [eLua][1] this may look like:
local chunk = loadfile("/rom/"..k..".lua")
-- put code into the table
self[k] = chunk;
return chunk;
end
})
Use it like this:
x, y, z = lib.foo(a, b, c)
With a bit of metatable magic, you could create a hierarchy too, and maybe collect functions into "modules" by loading all the leaves with a module.
Since your environment is very memory-constrained, I would suggest a convention over configuration approach for this as registering all available modules at load-time will surely affect memory footprint. Here is a suggestion:
local PATH = "/rom/src/";
local mt; mt = {
__mode = "v",
__index = function(self, k)
local fn = PATH .. self.__PATH .. k .. ".lua"
local chunk, errmsg = loadfile(fn) -- Warning Can cause security issues
if not chunk then
if io.open(fn) then
chunk = setmetatable({}, mt)
chunk.__PATH = self.__PATH .. k .. "/"
else
error(errmsg)
end
end
self[k] = chunk;
return chunk;
end
}
setmetatable(_G, mt)
Use like this:
local x, y, z = a.b.c(u, v, w) -- looks for "/rom/src/a/b/c.lua
In both variants, the Lua GC should automatically try to collect unused code (functions with reference count = 0) when memory gets tight.
More tips
Store code as byte-code. It loads faster and (usually) takes less space on flash, which makes it load even faster yet (But is usually not compatible between versions of Lua)
Keep variables as local as possible so Lua can clean up as quickly as possible
Avoid recursion
Avoid storing references to functions, long strings and tables in non-local variables (like _G)
Avoid tables
PS: Disclaimer All code in this answer is actually written from memory and thus not tested at all.
Reference:
[1] http://www.eluaproject.net/doc/v0.9/en_arch_romfs.html
Related
I was wondering if it would make sense to pass a meta table by reference vs declaring it in-line in setmetatable() when you want to use the same meta table for multiple tables.
My goal is to save memory, but only if it really makes a significant difference.
What I'm talking about is this:
-- Passing the meta table by reference:
JSON1 = {
metaTable = {
__index = function (t, k)
-- ...
end;
__call = function()
-- ...
end
};
parse = function(filePath)
local fakeParsedJson = {}
setmetatable(fakeParsedJson, JSON1.metaTable) -- Right here
return fakeParsedJson(filePath)
end;
}
VS
-- Passing the table in-line:
JSON2 = {
parse = function(filePath)
local fakeParsedJson = {}
setmetatable(fakeParsedJson, { -- Right here:
__index = function (t, k)
-- ...
end;
__call = function()
-- ...
end
})
return fakeParsedJson(filePath)
end;
}
I tried to find out if there is a significant difference in memory usage, but the only way I could find was to compare the gcinfo:
local start1 = gcinfo()
local example2_1 = JSON2.parse('example2_1.json')
local example2_2 = JSON2.parse('example2_2.json')
local example2_3 = JSON2.parse('example2_3.json')
local example2_4 = JSON2.parse('example2_4.json')
local example2_5 = JSON2.parse('example2_5.json')
print(gcinfo()-start1) -- Prints 1
local start2 = gcinfo()
local example1_1 = JSON1.parse('example1_1.json')
local example1_2 = JSON1.parse('example1_2.json')
local example1_3 = JSON1.parse('example1_3.json')
local example1_4 = JSON1.parse('example1_4.json')
local example1_5 = JSON1.parse('example1_5.json')
print(gcinfo()-start2) -- Prints 1
Here's my fiddle: https://repl.it/HfwS/34
It doesn't really look like there is a difference. But I just don't know what is actually happening under the hood.
When you call setmetatable(myTable,myMetaTable), will that write a complete copy of myMetaTable somewhere into myTable or will it just store a simple reference? Because if it would just store a reference, then it would make a lot of sense to have all my tables pointing to the same meta table.
(On x86_64, in Lua 5.3) every (empty) table costs 56 bytes. Every key/value entry in the table costs 32 bytes (but the number of entries is rounded up to the next power of two). (Byte counts may differ for different versions/platforms, but will be roughly the same +/- a power of two or so.)
If you have two entries in the metatable, that's 120 bytes per metatable. (You're also creating closures (function() … end), so it may actually be even more.)
Having the table constructor in argument position for the call to setmetatable means that every time that call is executed, a new independent table is created (+ new closures for the functions, …). (Also read the section on table constructors in the reference manual.) There is no smart compiler / no de-duplication / … In fact, there can't be, because other code could (potentially) modify a metatable, and then there's a clear semantic / observable difference between a single shared metatable and one metatable per thing. If that's not obvious, compare
Foo = { __name = "Foo", dump = print } ; Foo.__index = Foo
function newFoo( ) return setmetatable( { }, Foo ) end
and
function newFoo( )
local mt = { __name = "Foo", dump = print }
mt.__index = mt
return setmetatable( { }, mt )
end
If you say
t = { newFoo( ), newFoo( ), newFoo( ) }
getmetatable( t[1] ).dump = function( self ) print "<Foo>" end
for _, v in ipairs( t ) do v:dump( ) end
the first version will print
<Foo>
<Foo>
<Foo>
while the second one will print (e.g.)
<Foo>
Foo: 0x1267010
Foo: 0x1267120
which is clearly different behavior. So the compiler/… can not de-duplicate identical metatables, because other code (that was not yet seen) might modify one of the metatables, and then the observed behavior would be different.
▶ This means that if you create multiple (meta)tables, they must be kept somewhere. Storing several tables necessarily uses more memory than storing a single one, so having a table constructor in argument position for the call to setmetatable will use more memory than creating a table first and then passing a reference to it in the call.
That said, worrying about memory use should not be your primary concern. The semantics / "meaning" / observable behavior of your code are more important.
If you modify the metatable, should the behavior of all "objects" / values change? Or do you want to determine object types by metatable identity (getmetatable( x ) == Foo)? Then you must use a shared metatable (or an equivalent construction).
If you modify the metatable, should the behavior of only a single "object" change? Then you must construct & use separate metatables per "object" / value.
Only if you know that you will never modify the metatable, will not compare metatable references to determine object types, will not …, then these different approaches will show the same externally visible behavior, and only then you are free to choose based on secondary concerns (like memory usage, convenience / code brevity, …).
(In general, needing separately modifiable metatables is very rare, so using shared metatables (create first and pass the reference to setmetatable) is the usual approach – it saves memory and is nicer for debugging.)
Aside: gcinfo is very old and only returns integer approximations to the amount of memory used. Use collectgarbage "count" instead, and then you'll see a difference. (It returns kilobytes used, so multiply by 1024 to get bytes.)
Suppose I have the following two Lua files:
In a.lua:
local x = 5
f = dofile'b.lua'
f()
In b.lua:
local fun = function()
print(x)
end
return fun
Then if I run luajit a.lua in shell it prints nil since x cannot be seen in the function defined in b.lua. The expected printing should be 5. However if I put everything in a single file then it's exactly what I want:
In aa.lua:
local x = 5
local f = function()
print(x)
end
f()
Run luajit aa.lua it prints 5.
So why x cannot be seen in the first case?
As their name suggests, local variables are local to the chunk.
dofile() loads the chunk from another file. Since it's another chunk, it makes sense that the local variable x in the first chunk isn't seen by it.
I agree that it is somewhat unintuitive that this doesn't work.
You'd like to say, at any point in the code there is a clear set of variables that are 'visible' -- some may be local, some may be global, but there is some map that the interpreter can use to resolve names of either kind.
When you load a chunk using dofile, then it can see whatever global variables currently exist, but apparently it can't see any local variables. We know that 'dofile' is not like C/C++ inclusion macros, which would give exactly the behavior you describe for local variables, but still you might reasonably expect that this part of it would work the same.
Ultimately there's no answer but "that's just not how they specified the language". The only satisfying answer is probably along the lines 'because otherwise it would cause non-obvious problem X' or 'because then use-case Y would go slower'.
I think the best answer is that, if all names were dynamically rebound according to the scope in which they are loaded when you use loadfile / dofile, that would inhibit a lot of optimization and such when compiling chunks into bytecode. In the lua system, name resolution works like 'either it is local in this scope, and then it binds to that (known) object, or, it is a lookup in the (unique) global table.' This system is pretty simple, there are only a few options and not a lot of room for complexity.
I don't think that running byte code even keeps track of the names of local variables, it discards them after the chunk is compiled. They would have to undo that optimization if they wanted to allow dynamic name resolution at chunk loading time like you suggest.
If your question is not really why but how can I make it work anyways, then one way you can do it is, in the host script, put any local variables that you want to be visible in the environment of the script that is called. When you do this you need to split dofile into a few calls. It's slightly different in lua 5.1 vs lua 5.2.
In lua 5.1:
In a.lua:
local shared = { x = 5 }
temp = loadfile('b.lua')
setfenv(temp, shared)
f = temp()
f()
In lua 5.2:
In a.lua:
local shared = { x = 5 }
temp = loadfile('b.lua', 't', shared)
f = temp()
f()
The x variable defined in module a.lua cannot be seen from b.lua because it was declared as local. The scope of a local variable is its own module.
If you want x to be visible from b.lua, just need to declare it global. A variable is either local or global. To declare a variable as global, just simply do not declare it as local.
a.lua
x = 5
f = dofile'b.lua'
f()
b.lua
local fun = function()
print(x)
end
return fun
This will work.
Global variables live within the global namespace, which can be accessed at any given time via the _G table. When Lua cannot solve a variable, because it's not defined inside the module where is being used, Lua searches that variable in the global namespace. In conclusion, it's also possible to write b.lua as:
local fun = function()
print(_G["x"])
end
return fun
I'd like to be able to make a program for ComputerCraft using MoonScript, but due to the way that CC is sandboxed to prevent security issues on Minecraft servers and such, I can't require moonscript directly and run moonscript code from there. I have to convert my moonscript code to lua.
This is problematic, however, due to the fact that the class implementation for moonscript is very big, and not very filesize-conservative. When I type "class Bacon", it outputs this for lua:
local Bacon
do
local _parent_0 = nil
local _base_0 = { }
_base_0.__index = _base_0
if _parent_0 then
setmetatable(_base_0, _parent_0.__base)
end
local _class_0 = setmetatable({
__init = function(self, ...)
if _parent_0 then
return _parent_0.__init(self, ...)
end
end,
__base = _base_0,
__name = "Bacon",
__parent = _parent_0
}, {
__index = function(cls, name)
local val = rawget(_base_0, name)
if val == nil and _parent_0 then
return _parent_0[name]
else
return val
end
end,
__call = function(cls, ...)
local _self_0 = setmetatable({}, _base_0)
cls.__init(_self_0, ...)
return _self_0
end
})
_base_0.__class = _class_0
if _parent_0 and _parent_0.__inherited then
_parent_0.__inherited(_parent_0, _class_0)
end
Bacon = _class_0
return _class_0
end
And this is for every class implementation, which is kind of ridiculous. Is there any way I can shorten this in my moonscript code?
The amount of generated code has been cleaned up quite a bit in the latest version, 0.2.4: http://leafo.net/posts/moonscript_v024.html#code_generation_changes
A class now minimally looks like this
local Hello
do
local _base_0 = { }
_base_0.__index = _base_0
local _class_0 = setmetatable({
__init = function() end,
__base = _base_0,
__name = "Hello"
}, {
__index = _base_0,
__call = function(cls, ...)
local _self_0 = setmetatable({}, _base_0)
cls.__init(_self_0, ...)
return _self_0
end
})
_base_0.__class = _class_0
Hello = _class_0
end
Just looking at the code I can remove some dead paths due to _parent_0 being nil...
local Bacon
do
local _base_0 = { }
_base_0.__index = _base_0
local _class_0 = setmetatable({
__init = function(self, ...)
end,
__base = _base_0,
__name = "Bacon",
}, {
__index = function(cls, name)
return rawget(_base_0, name)
end,
__call = function(cls, ...)
local _self_0 = setmetatable({}, _base_0)
cls.__init(_self_0, ...)
return _self_0
end
})
_base_0.__class = _class_0
Bacon = _class_0
return _class_0
end
You can find a static analyser to do this for you.
Otherwise if it's purely code size (in bytes) that concerns you, then use a compressor (e.g. Squish)
There is no way that you can shorten this in your moonscript code. But you might be able to post-process the output manually rather trivially to refactor common portions. Is it worth doing? Consider this:
If you want to use the notion of "class" as it is available in moonscript, then you have to code all those steps anyways (otherwise you get a "class" with different "class" behavior).
The difference is that in moonscript, N classes cause N copies of the code you posted, differing only by two lines each, whereas if you implemented moonscript's notion of class manually, you would probably use a function to define classes, as is done in some Lua OO libraries: Bacon = newClass("Bacon") would call the above code, with the "__name =" and "Bacon =" lines replaced appropriately.
So by copying moonscript code instead of code it manually, each class requires 20 lines of code (latest moonscript). But unless your classes are trivial, it is likely that they will contain a fair bit more code than 20 lines, probably 200.
So while it is true that you are duplicating 18 lines per class, this is likely less than 10% of the code you will have to write.
But that is an assumption. If the assumption is wrong, then the question is, why do you need classes in the first places? Just use basic Lua objects based on tables with ':' sugar.
If you really want use the moonscript output, I think your only option is to refactor it manually. The code you posted can be put in a function and only two lines need to be changed to parametrize for class name.
Note that there several advantages to leaving the Lua code duplication generated by moonscript in your Lua files, at least for oft-used operations (class definition is probably not one of them, but moonscript does way more than that):
it cuts back on function calls; this is likely to increase speed since no need to push variables onto a stack, call a function, put return values onto the stack, read stack, pop values off stack). Whether the speed increase is noticeable will depend a lot on how often the code is called.
it is more maintainable: if done right, you can likely update the code when moonscript gets updated more easily.
I just started with Lua as part of a school assignment. I'd like to know if there's an easy way to implement profiling for Lua? I need something that displays allocated memory, variables in use regardless of their type, etc.
I've been finding C++ solutions which I have been able to compile successfully but I don't know how to import them to the Lua environment.
I also found Shinny but I couldn't find any documentation about how to make it work.
There are several profilers available that you can check, but most of them target execution time (and are based on debug hook).
To track variables, you will need to use debug.getlocal and debug.getupvalue (from your code or from debug hook).
To track memory usage you can use collectgarbage(count) (probably after collectgarbage(collect)), but this only tells you total memory in use. To track individual data structures, you may need traverse global and local variables and calculate the amount of space they take. You can check this discussion for some pointers and implementation details.
Something like this would be the simplest profiler that tracks function calls/returns (note that you should not trust absolute numbers it generates, only relative ones):
local calls, total, this = {}, {}, {}
debug.sethook(function(event)
local i = debug.getinfo(2, "Sln")
if i.what ~= 'Lua' then return end
local func = i.name or (i.source..':'..i.linedefined)
if event == 'call' then
this[func] = os.clock()
else
local time = os.clock() - this[func]
total[func] = (total[func] or 0) + time
calls[func] = (calls[func] or 0) + 1
end
end, "cr")
-- the code to debug starts here
local function DoSomethingMore(x)
x = x / 2
end
local function DoSomething(x)
x = x + 1
if x % 2 then DoSomethingMore(x) end
end
for outer=1,100 do
for inner=1,1000 do
DoSomething(inner)
end
end
-- the code to debug ends here; reset the hook
debug.sethook()
-- print the results
for f,time in pairs(total) do
print(("Function %s took %.3f seconds after %d calls"):format(f, time, calls[f]))
end
I'm just getting started with Lua. In the example I'm learning from (the Ghosts & Monsters Corona open source), I see this pattern repeatedly.
local director = require("director")
local mainGroup = display.newGroup()
local function main()
mainGroup:insert(director.directorView)
openfeint = require ("openfeint")
openfeint.init( "App Key Here", "App Secret Here", "Ghosts vs. Monsters", "App ID Here" )
director:changeScene( "loadmainmenu" )
return true
end
main()
Is this some sort of convention experienced Lua programmers recommend or are there genuine advantages to doing it this way? Why wouldn't you just skip the function all together and do this:
local director = require("director")
local mainGroup = display.newGroup()
mainGroup:insert(director.directorView)
local openfeint = require ("openfeint")
openfeint.init( "App Key Here", "App Secret Here", "Ghosts vs. Monsters", "App ID Here" )
director:changeScene( "loadmainmenu" )
Is there some implicit benefit to the first style over the second? Thanks!
Is this some sort of convention experienced Lua programmers recommend or are there genuine advantages to doing it this way?
It's not typical. The advantage is that object state is private, but that's not enough of an advantage to recommend it.
I see this pattern repeatedly.
I've never seen it before, and it happens only once in the source you posted.
EDIT: Adding a response to a question asked in the comments below this post.
A function which accesses external local variables binds to those variables and is called a 'closure'. Lua (for historical reasons) refers to those bound variables as 'upvalues'. For example:
local function counter()
local i = 1
return function()
print(i)
i = i + 1
end
end
local a, b = counter(), counter()
a() a() a() b() --> 1 2 3 1
a and b are closures bound to different copies of i, as you can see from the output. In other words, you can think of a closure as function with it's own private state. You can use this to simulate objects:
function Point(x,y)
local p = {}
function p.getX() -- syntax sugar for p.getX = function()
return x
end
function p.setX(x_)
x = x_
end
-- for brevity, not implementing a setter/getter for y
return p
end
p1 = Point(10,20)
p1.setX(50)
print(p1.getX())
Point returns a table of closures, each bound to the locals x and y. The table doesn't contain the point's state, the closures themselves do, via their upvalues. An important point is that each time Point is called it creates new closures, which is not very efficient if you have large quantities of objects.
Another way of creating classes in Lua is to create functions that take a table as the first argument, with state being stored in the table:
function Point(x,y)
local p = {x=x,y=y}
function p:getX() -- syntax sugar for p.getX = function(self)
return self.x
end
function p:setX(x)
self.x = x
end
return p
end
p1 = Point(10,20)
p1:setX(50) -- syntax sugar for p1.setX(p1, 50)
print(p1:getX()) -- syntax sugar for p1.getX(p1)
So far, we're still creating new copies of each method, but now that we're not relying on upvalues for state, we can fix that:
PointClass = {}
function PointClass:getX() return self.x end
function PointClass:setX(x) self.x = x end
function Point(x,y)
return {
x = x,
y = y,
getX = PointClass.getX,
setX = PointClass.getY,
}
end
Now the methods are created once, and all Point instances share the same closures. An even better way of doing this is to use Lua's metaprogramming facility to make new Point instances automatically look in PointClass for methods not found in the instance itself:
PointClass = {}
PointClass.__index = PointClass -- metamethod
function PointClass:getX() return self.x end
function PointClass:setX(x) self.x = x end
function Point(x,y)
return setmetatable({x=x,y=y}, PointClass)
end
p1 = Point(10,20)
-- the p1 table does not itself contain a setX member, but p1 has a metatable, so
-- when an indexing operation fails, Lua will look in the metatable for an __index
-- metamethod. If that metamethod is a table, Lua will look for getX in that table,
-- resolving p1.setX to PointClass.setX.
p1:setX(50)
This is a more idiomatic way of creating classes in Lua. It's more memory efficient and more flexible (in particular, it makes it easy to implement inheritance).
I frequently write my own Lua scripts this way because it improves readability in this case:
function main()
helper1( helper2( arg[1] ) )
helper3()
end
function helper1( foo )
print( foo )
end
function helper2( bar )
return bar*bar
end
function helper3()
print( 'hello world!' )
end
main()
This way the "main" code is at the top, but I can still define the necessary global functions before it executes.
A simple trick, really. I can't think of any reason to do this besides readability.
The first style could be used too improove readability, but I would rather give the function some meaningful name instead of main or just go without the function.
By the way, I think it's always a good practice to name blocks of code, i.e. put them into functions or methods. It helps explain your intend with that piece of code, and encourages reuse.
I don't see much point to the first style as you've shown it. But if it said something like if arg then main() end at the bottom, the script might (just might) be useful as a loadable "library" in addition to being a standalone script. That said, having a main() like that smacks of C, not Lua; I think you're right to question it.