I have a question about how closures are implemented.
Say this is in a file named test.lua:
local a = 'asdf'
local function b()
return a
end
a = 10
return b
And another file does
a = require 'test'
a()
it will print
10
If a is a pointer on the stack to 'asdf' (on the heap I assume, but it doesn't matter), and the closure b is created so presumably the address that was in a is saved for b to use, how does a = 10 change the pointer inside the closure as well?
Wikipedia says quite well what is perplexing me:
A language implementation cannot easily support full closures if its run-time memory model allocates all local variables on a linear stack1. In such languages, a function's local variables are deallocated when the function returns.
I was thinking that perhaps b really didn't save a pointer to 'asdf' but a stack offset to a, so that you can change a and the stack offset will get you to a which points to the last thing you set a to, but then how does this work when a (the pointer) is popped off the stack and the stack offset becomes invalid?
1 I know Lua doesn't allocate the values on the stack, but it allocates local pointers on the stack to values in the heap, doesn't it?
I really wish you had named these variables a bit more reasonably. So I will:
local inner = 'asdf'
local function b()
return inner
end
inner = 10
return b
and
func = require 'test'
func()
OK, now that we know what we're talking about, I can proceed.
The Lua chunk test has a local variable called inner. Within that chunk you create a new function b. Since this is a new function, it has a scope within the scope of the chunk test.
Since it is within a function, it has the right to access local variables declared outside of that function. But because it is inside of a function, it does not access those variables like it would one of its own locals. The compiler detects that inner is a local variable declared outside of the function's scope, so it converts it into what Lua calls an "upvalue".
Functions in Lua can have an arbitrary number of values (up to 255) associated with them, called "upvalues". Functions created in C/C++ can store some number of upvalues by using lua_pushcclosure. Functions created by the Lua compiler use upvalues to provide lexical scoping.
A scope is everything that happens within a fixed block of Lua code. So:
if(...) then
--yes
else
--no
end
The yes block has a scope, and the no block has a different scope. Any local variables declared in the yes block cannot be accessed from the no block, because they are outside of the scope of the no block.
The Lua constructs that define a scope are if/then/else/end, while/do/end, repeat/until, do/end, for/end, and function/end. Also, each script, called a Lua "chunk", has a scope.
Scopes are nested. From within one scope, you can access local variables declared in a higher scope.
A "stack" represents all variables declared as local within a particular scope. So if you have no local variables in a certain scope, the stack for that scope is empty.
In C and C++, the "stack" that you are familiar with is just a pointer. When you call a function, the compiler has predetermined how many bytes of space that the function's stack needs. It advances the pointer by that amount. All stack variables used in the function are just byte offsets from the stack pointer. When the function exits, the stack pointer is decreased by the stack amount.
In Lua, things are different. The stack for a particular scope is an object, not merely a pointer. For any particular scope, there are some number of local variables defined for it. When the Lua interpreter enters a scope, it "allocates" a stack of the size necessary to access those local variables. All references to local variables are just offsets into that stack. Access to local variables from higher scopes (previously defined) simply access a different stack object.
So in Lua, you conceptually have a stack of stacks (which I will refer to as the "s-stack" for clarity). Each scope creates a new stack and pushes it, and when you leave a scope, it pops the stack off of the s-stack.
When the Lua compiler encounters a reference to a local variable, it converts that reference into an index into the s-stack, and an offset into that particular stack. So if it accesses a variable in the current local stack, the index into the s-stack refers to the top of the s-stack, and the offset is the offset into that stack where the variable is.
That's fine for most Lua constructs that access scopes. But function/end don't just create a new scope; they create a new function. And this function is allowed to access stacks that aren't just the local stack of that function.
Stacks are objects. And in Lua, objects are subject to garbage collection. When the interpreter enters a scope, it allocates a stack object and pushes it. So long as the stack object is pushed onto the s-stack, it cannot be destroyed. The stack of stacks refers to the object. However, once the interpreter exits the scope, it pops the stack off of the s-stack. So since it is no longer referenced, it is subject to being collected.
However, a function that accesses variables outside of its own local scope can still be referencing that stack. When the Lua compiler sees a reference to a local variable that is not within the function's local scope, it alters the function. It figures out which stack the local it is referencing belongs to, and then stores that stack as an upvalue in the function. It converts the reference to that variable to an offset into that particular upvalue, rather than an offset into a stack that is currently on the s-stack.
So as long as the function object continues to exist, so too will the stack(s) that it references.
Remember that stacks are dynamically created and destroyed as the Lua interpreter enters and exits the scope of functions. So if you were to run test twice, by calling loadfile and executing the returned function twice, you would get two separate functions that refer to two separate stacks. Neither function will see the value from the other.
Note that this may not be exactly how it's implemented, but that's the general idea behind it.
Related
I'm trying to build my first ever CHIP-8 emulator from scratch using C. While writing necessary code for the instructions, I came across this opcode:
00EE - RET
Return from a subroutine.
The interpreter sets the program counter to the address at the top of the stack, then subtracts 1 from the stack pointer.
(http://devernay.free.fr/hacks/chip8/C8TECH10.HTM)
I know that a subroutine is basically a function, but what does it mean to 'return' from a subroutine? And what is happening to the program counter, stack, and the stack pointer respectively?
(One additional question): If I created an array that can hold 16 values to represent the stack, will the 'top of the stack' be STACK[0] or STACK[15]? And where should my stack pointer be?
To return from a subroutine is to return code execution to the point it was at before the subroutine was called.
Therefore, given that calling a subroutine pushes the current address PC+2 (+2 to jump past the call instruction) onto the stack. Returning from a subroutine will return execution to the address that was pushed to the stack by popping the address from the stack. (e.g. pc=stack[sp]; sp-=2;)
As for the additional question, it really depends on whether you define your stack as being ascending or descending. For the CHIP-8 the choice is not specified.
I read that it is faster and better to keep most of your functions local instead of global.
So I'm doing this:
input = require("input")
draw = require("draw")
And then in input.lua for example:
local tableOfFunctions = {isLetter = isLetter, numpadCheck = numpadCheck, isDigit = isDigit, toUpper = toUpper}
return tableOfFunctions
Where isLetter, numpadCheck etc are local functions for that file.
Then I call the functions like so:
input.isLetter(key)
Now, my question is: Am I reinventing the wheel with this? Aren't global functions stored in a lua table? I do like the way it looks with the input. before the function name, keeps it nice and tidy so I may keep it if it's not a bad coding practise.
Reinventing the wheels tailored to your personal needs is centerpiece of lua.
The method you describe is described as a valid one by lua creator himself in his book, here.
Everything in Lua is stored inside a table. The "faster" local function (as well as faster local variables) comes from the way of how globals and upvalues are looked up.
Below the line there's a quote of relevant part of more detailed explanation on speed that happened to occur in on game's forum.
Apart from that, locals are recommended due to cleanness of the code and error proofing.
In lua a table is created with {}, this operator reserves a certain amount of memory in the ram for the table. That reserved space stays constant and unmovable, exceptions are implementation details that script writer should not concern himself with.
Any variable you assign table to
a={};
b={ c={a} }
is just a pointer to the table in memory. A pointer takes up either 32 or 64 bits and that's it.
Whenever you pass table around only those 64 bits are copied.
Whenever you query a table in a table:
return b.c[1]
computer follows the pointer stored in b, finds a table object in ram, queries it for key "c", takes pointer to another table, queries that for key "1" then returns the pointer to the table a. Just a simple pointer hopping, workload on par with arithmetic.
Every function has associated table _ENV, any variable lookup
return a
is actually a query to that table
return _ENV.a
If the variable is local, it is stored in _ENV.
If there's no variable in _ENV with the given name, then global variables are queried, those actually reside in top-level table, the _ENV of the root function of the script (it is require or dofile function that loads and executes the script).
Usually, a link to the global table is stored in any other _ENV as _G. So the access to a global variable
return b
is actually something like
return _ENV.b or _ENV._G.b
Thus it is about 3 pointer jumps instead of 1.
Here is convoluted example that should give you an insight on the amount of work that implies:
%RUN THIS IN STANDALONE LUA INTERPRETER
local depth=100--how many pointers will be in a chain
local q={};--a table
local a={};--a start of pointer chain
local b=a; -- intermediate variable
for i=1,depth do b.a={} b=b.a end; --setup chain
local t=os.clock();
print(q)
print(os.clock()-t);--time of previous line execution
t=os.clock(); --start of pointer chain traversal
b=a
while b.a do b=b.a end
print(b)
print(os.clock()-t)--time of pointer traversal
When a pointer chain is about 100 elements, system load fluctuations may actually cause the second time be smaller. The direct access gets notably faster only when you change depth to thousands and more intermediate pointers.
Note that, whenever you query uninitialized variable, all 3 jumps are taken.
Globals are stored in the reserved table _G (the contents of which you can examine at any time), but it is good programming practice to avoid the use of globals.
Unless there is a very good reason not to, your table input should be local as well.
From Programming in Lua:
It is good programming style to use local variables whenever possible. Local variables help you avoid cluttering the global environment with unnecessary names. Moreover, the access to local variables is faster than to global ones.
In Go, it seems there are no constructors, but it is suggested that you allocate an object of a struct type using a function, usually named by "New" + TypeName, for example
func NewRect(x,y, width, height float) *Rect {
return &Rect(x,y,width, height)
}
However, I am not sure about the memory layout of Go. In C/C++, this kind of code means you return a pointer, which point to a temporary object because the variable is allocated on the stack, and the variable may be some trash after the function return. In Go, do I have to worry such kind of thing? Because It seems no standard shows that what kind of data will be allocated on the stack vs what kind of data will be allocated on the heap.
As in Java, there seems to have a specific point out that the basic type such as int, float will be allocated on the stack, other object derived from the object will be allocated on the heap. In Go, is there a specific talk about this?
The Composite Literal section mentions:
Taking the address of a composite literal (§Address operators) generates a unique pointer to an instance of the literal's value.
That means the pointer returned by the New function will be a valid one (allocated on the stack).
Calls:
In a function call, the function value and arguments are evaluated in the usual order.
After they are evaluated, the parameters of the call are passed by value to the function and the called function begins execution.
The return parameters of the function are passed by value back to the calling function when the function returns.
You can see more in this answer and this thread.
As mentioned in "Stack vs heap allocation of structs in Go, and how they relate to garbage collection":
It's worth noting that the words "stack" and "heap" do not appear anywhere in the language spec.
The blog post "Escape Analysis in Go" details what happens, mentioning the FAQ:
When possible, the Go compilers will allocate variables that are local to a function in that function's stack frame.
However, if the compiler cannot prove that the variable is not referenced after the function returns, then the compiler must allocate the variable on the garbage-collected heap to avoid dangling pointer errors.
Also, if a local variable is very large, it might make more sense to store it on the heap rather than the stack.
The blog post adds:
The code that does the “escape analysis” lives in src/cmd/gc/esc.c.
Conceptually, it tries to determine if a local variable escapes the current scope; the only two cases where this happens are when a variable’s address is returned, and when its address is assigned to a variable in an outer scope.
If a variable escapes, it has to be allocated on the heap; otherwise, it’s safe to put it on the stack.
Interestingly, this applies to new(T) allocations as well.
If they don’t escape, they’ll end up being allocated on the stack. Here’s an example to clarify matters:
var intPointerGlobal *int = nil
func Foo() *int {
anInt0 := 0
anInt1 := new(int)
anInt2 := 42
intPointerGlobal = &anInt2
anInt3 := 5
return &anInt3
}
Above, anInt0 and anInt1 do not escape, so they are allocated on the stack;
anInt2 and anInt3 escape, and are allocated on the heap.
See also "Five things that make Go fast":
Unlike C, which forces you to choose if a value will be stored on the heap, via malloc, or on the stack, by declaring it inside the scope of the function, Go implements an optimisation called escape analysis.
Go’s optimisations are always enabled by default.
You can see the compiler’s escape analysis and inlining decisions with the -gcflags=-m switch.
Because escape analysis is performed at compile time, not run time, stack allocation will always be faster than heap allocation, no matter how efficient your garbage collector is.
tl;dr version: is a garbage collection metamethod safe during a Lua state shutdown if it accesses global variables? What about local up-values?
When a lua_State is closed with lua_close, the Lua documentation says that all of the objects are deleted. And it says that any associated garbage-collection metamethods are guaranteed to be called.
Great.
However, there are two possible use cases of GC metamethods that are uncertain under this paradigm:
What if you have a GC metamethod that uses a local variable that stores a collectable value. Say, a string, a function, etc. That is, your GC metamethod is defined as this:
local some_string = "string"
function mt:__gc() --[[Do something with some_string]] end
What happens in this case? Is it possible for some_string to be collected? I know that if the object the metatable is on is being collected under normal circumstances, then this isn't possible. Lua would guarantee that the value of some_string would remain until the GC function itself is collected.
But since all objects are being destroyed by lua_close, is it possible that upvalues of the GC function could be destroyed before the function is? I don't think it is (as this could cause all kinds of problems), but I'm looking for the real answer, not what I think.
I admit that it is unlikely that #1 will be a problem, as it would create many issues with GC metamethods. However, this is a completely different matter:
local some_string = "string"
function mt:__gc() print(some_string) end
This looks like #1, but it isn't. Why? Because it accesses a global variable. Namely, print.
There is no direct association between the function and whatever value is stored in print (unlike case 1, where some_string is clearly an upvalue of the function). As such, it is possible for print to be collected before calling the function during Lua state shutdown.
The question is this: is it safe for a garbage collection metamethod to ever use anything from the global table (ignoring the possibility of deliberate sabotage of the global table vis-a-vis print = nil) during Lua state shutdown? Or, as a general rule, should they always explicitly make any functions or data they touch local? Will that be sufficient to avoid problems?
Any data that is accessible from the __gc method is safe to access; this includes both locals and the tables that are accessible, like _G, which contains print.
What can be a problem are (un-)loaded C libraries; this affects Lua 5.1, and is fixed in Lua 5.2.1. See the patch for "Finalizers may call functions from a dynamic library after the library has been unloaded".
Is a z/OS PL/I CONTROLLED variable preserved between separate invocations of a procedure? Let’s suppose that we need a counter that is internal to a subroutine and preserved across invocations. The easiest way to do it would be to use a static variable initialized to zero and incremented on each entry to the subroutine. But you can’t do that if the program has to be reentrant. So the question is whether we have access to a controlled variable that was allocated in a previous call. Would the following code work?
PROC1: PROCEDURE OPTIONS(MAIN);
...
CALL A;
...
A: PROCEDURE;
DECLARE COUNT CONTROLLED ALIGNED FIXED BIN(15);
IF (ALLOCATION(COUNT) = 0)
THEN ALLOCATE COUNT INIT(1);
ELSE COUNT = COUNT + 1;
...
END A;
END PROC1;
According to the PL/I Language Reference, after you ALLOCATE a variable, you do not need to FREE it (though that is generally good practice), and “All controlled storage is freed at the end of the program.” It doesn’t say that storage is freed at the end of the block.
The PL/I Programming Guide provides some clues in the chapter Using PLIDUMP in the Locating Controlled Variables section, but it is not definitive. It says that the key to locating a controlled variable is to find its anchor. With NORENT WRITABLE there is an anchor in static storage. With NORENT NOWRITABLE(FWS) there is an address to an anchor automatic storage. (There is an extra level of indirection.) With NORENT NOWRITABLE(PRV) there appears to be a static table with an offset into a private table for each controlled variable. In other words, depending on the processing options, maybe the variable is accessible, and maybe it isn’t. It doesn’t say anything about using the RENT option.
Any thoughts?
As per the PL/I Programming Guide compile time option "RENT", Your code is "naturally reentrant" if it does not alter any of its static variables.
The RENT option specifies that the compiler is to take code that is not naturally reentrant and make it reentrant.
Thus, you can increment STATIC variable on each entry to subroutine if the program is compiled with RENT option.
Please refer to this link => Rent Option from PL/I Programming Guide
As per "PL/I Structured Programming" by J.K. Hughes, A REENTRANT procedure may be called by other procedures asynchronously. For example task B invokes SQRT function. While this function is in process of computing square root, task A (having higher execution priority than task B) needs to gain control of the system and use SQRT function. The SQRT function is interrupted and intermediate results for task B saved; then task A uses SQRT function.
When task A completes its execution, control is returned to task B at the point where it was interrupted. Then, task B completes its use of SQRT function.