Is there a way to run Lua code from a C/C++ program at a more fine-grained level than a standard "lua_pcall" function call? Ideally I'd like to be able to loop over a list of low-level bytecode instructions (assuming it has such things) and run them one by one, so that I could write my own scheduler which had more control of things than just running a complete Lua function from start to finish.
The reason I want to do this is because I wish to implement C functions which Lua code can call which would cause the program to wait until a certain (potentially long-winded) action had completed before continuing execution. There would be a high proportion of such function calls in a typical Lua script, so the idea of rewriting it to use callbacks once the action has completed isn't really practical.
Perhaps side-stepping the question, but you could use Lua coroutines rather than custom C stuff to wait until some event occurs.
For example, one coroutine could call a waitForEvent() function. In there, you can switch to another coro until that event occurs, then resume the first one. Take a look at the lua coro docs for more about that.
Jder's suggestion to use coroutines will work very well if you can write those long waiting C routines using Lua's cooperative threading (explicit yield) feature. You'll still use lua_pcall() to enter Lua, but the entry point will be your coroutine manager function.
This only works though if the C routines don't do anything while they wait. If they are long running because they calculate something for example, then you need to run multiple OS threads. Lua is threadsafe -- just create multiple threads and run lua_open() in each thread.
From http://www.lua.org/pil/24.1.html
The Lua library defines no global
variables at all. It keeps all its
state in the dynamic structure
lua_State and a pointer to this
structure is passed as an argument to
all functions inside Lua. This
implementation makes Lua reentrant and
ready to be used in multithreaded
code.
You can also combine the two approaches. If you have a wrapper Lua function to start an OS thread, you can yield after you start the thread. The coroutine manager will keep track of threads and continue a coroutine when the thread it started has finished. This lets you use a single Lua interpreter with multiple worker threads running pure C code.
If you go the OS threading way, please have a look at Lua Lanes. I would see it the perfect solution to what you're trying to achieve (= throw one addon module to the mix and you'll be making clear, understandable and simple code with multithreading seamlessly built in).
Please tell us how your issue got solved. :)
Does the debugging interface help?
Related
I have written a code that runs multiple tasks one after the other in Lua. There is simply a loop that calls each function and then use the output for the next one and so on.
Usually there is no issue, but it happens that there is some error. In these situations the code would just stop working.
What I want is to monitor my code. In case there is an error at some point, I can call plan B and avoid that the program interrupts.
I've read about Lua coroutines, and it appears to be what i am looking for. If there is an error, the coroutine returns the error.
My question is whether I can stop coroutine that last too long. Lua offers asynchronous coroutines, so I thought that I cannot execute a function and simultaneously compute how long it has been busy doing the task. Am I getting this right, or is there a way to basically stop coroutines from the outside with some condition ?
Thank you
This is not possible using Lua coroutines. Lua coroutines are cooperative; they do not (inherently) support preemption, which is what you're looking for. For a coroutine to return execution to the caller of coroutine.resume, it has to "cooperate" by calling coroutine.yield or error.
As has been pointed out in the comments, it is possible to set a line hook - or even an "instruction hook" - to implement preemption of Lua-only code, by interrupting the code on every new line / every couple instructions executed and checking whether a timeout is met.
This stops working as soon as you call a (blocking) C function, including Lua standard library functions; it only works well for "pure Lua" code which only does constant-time API calls, since the API call is just a single line of code / instruction for Lua; Lua has no view "into" the function.
To solve this issue, you'll need actual multithreading e.g. using lualanes, multiprocessing (e.g. using luaposix's fork, or a nonblocking API and polling.
How to wait all coroutine finished in lua just like waitgroup in go? Waitgroup in go is completed with Semaphobe. But how to complete it in lua? Or any better scheme?
Lua is not a threaded scripting language. It has no concept of multiple things being able to happen at once (you can have multiple independent Lua states executing on distinct threads, but they're different Lua states. If you want them to be able to talk to each other, then you have to provide the means to do so manually).
As such, a Lua coroutine is simply a function which can choose to suspend its execution, returning some number of values to the function which invoked the coroutine. The user can then resume the coroutine, allowing it to further perform some task. This is cooperative multitasking; a task suspends execution only when it chooses to. This is in contrast to pre-emptive multitasking, where a task can just stop executing or execute in parallel with something else.
Given this, the idea of "waiting" for a coroutine or group of coroutines simply doesn't make sense. You resume a coroutine when you want it to execute further; it's not a passive action.
You could certainly create a list of coroutines and resume each one in turn repeatedly until all of them have terminated. But there's no Lua standard library function to do that, because it's generally not useful for the uses coroutines perform.
I've been looking at lua and lvm.c. I'd very much like to implement an interface to allow me to control the VM interpreter state.
Cooperative multitasking from within lua would not work for me (user contributed code)
The debug hook gets me only about 50% of the way there, instruction execution limits, but it raises an exception which just crashes the running lua code - but I need to be able to tweak it even further.
I want to create a system where 10's of thousands of lua user scripts are running - individual threads would not work, and the execution limits would cause headache for beginning developers, I'm going to control execution speeds too. but ultimately
while true do
end
will execute forever, and I really don't care that it is.
Any ideas, help or other implementations that I could look at?
EDIT: This is not about sandboxing pretend I'm an expert in that field for this conversation
EDIT: I do not want to use an internally ran lua code coroutine based controller.
EDIT: I want to run one thread, and manage a large number of user contributed lua scripts, an external process level control mechansim would not scale at all.
You can search for Lua Sandbox implementations; for example, this wiki page and SO question provide some pointers. Note that most of the effort in sandboxing is focused on not allowing you to execute bad code, but not necessarily on preventing infinite loops. For better control you may need to combine Lua sandboxing with something like LXC or cpulimit. (not relevant based on the comments)
If you are looking for something Lua-based, lightweight, but not necessarily 100% foolproof, then you can try running your client code in a separate coroutine and set a debug hook on that coroutine that will be triggered every N-th line. In that hook you can check if the process you are running exceeded its quotes. You also need to take care of new coroutines started as those need to have their own hooks set (you either need to disable coroutine.create/wrap or to replace them with something that sets the debug hook you need).
The code in this case may look like:
local coro = coroutine.create(client_func)
debug.sethook(coro, debug_hook, "l", 1000) -- trigger hook on every 1000th line
It's not foolproof, because it may block on some IO operation and the debug hook will not help there.
[Edit based on updated question and comments]
Between "no lua code coroutine based controller" and "no external process control mechanism" I don't think you are left with much choice. It may be that your only option is to run one VM per user script and somehow give ticks to those VMs (there was a recent question on SO on this, but I can't find it). Before going this route, I would still try to do this with coroutines (which should scale to tens of thousands easily; Tir claims supporting 1M active users with coroutine-based architecture).
The mechanism would roughly look like this: you install the debug hook as I shown above and from that hook you yield back to your controller, which then decides what other coroutine (user script) to resume. I have this very mechanism working in the Lua debugger I've been developing (although it only does it for one client script). This doesn't protect you from IO calls that can block and for that you may still need to have a watchdog at the VM level to see if it's been blocked for longer than needed.
If you need to serialize and deserialize running code fragments that preserve upvalues and such, then Pluto is probably your only option.
Look at implementing lua_lock and lua_unlock.
http://www.lua.org/source/5.1/llimits.h.html#lua_lock
Take a look at lulu. It is lua VM written on lua. It's for Lua 5.1
For newer version you need to do some work. But it's then you really can make a schelduler.
Take a look at this,
https://github.com/amilamad/preemptive-task-scheduler-for-lua
I maintain this project. It,s a non blocking preemptive scheduler for running lua code. Suitable for long running game scripts.
Recently, I have encountered many difficulties when I was developing using C++ and Lua. My situation is: for some reason, there can be thousands of Lua-states in my C++ program. But these states should be same just after initialization. Of course, I can do luaL_loadlibs() and lua_loadfile() for each state, but that is pretty heavy(in fact, it takes a rather long time for me even just initial one state). So, I am wondering the following schema: What about keeping a separate Lua-state(the only state that has to be initialized) which is then cloned for other Lua-states, is that possible?
When I started with Lua, like you I once wrote a program with thousands of states, had the same problem and thoughts, until I realized I was doing it totally wrong :)
Lua has coroutines and threads, you need to use these features to do what you need. They can be a bit tricky at first but you should be able to understand them in a few days, it'll be well worth your time.
take a look to the following lua API call I think it is what you exactly need.
lua_State *lua_newthread (lua_State *L);
This creates a new thread, pushes it on the stack, and returns a pointer to a lua_State that represents this new thread. The new thread returned by this function shares with the original thread its global environment, but has an independent execution stack.
There is no explicit function to close or to destroy a thread. Threads are subject to garbage collection, like any Lua object.
Unfortunately, no.
You could try Pluto to serialize the whole state. It does work pretty well, but in most cases it costs roughly the same time as normal initialization.
I think it will be hard to do exactly what you're requesting here given that just copying the state would have internal references as well as potentially pointers to external data. One would need to reconstruct those internal references in order to not just have multiple states pointing to the clone source.
You could serialize out the state after one starts up and then load that into subsequent states. If initialization is really expensive, this might be worth it.
I think the closest thing to doing what you want that would be relatively easy would be to put the states in different processes by initializing one state and then forking, however your operating system supports it:
http://en.wikipedia.org/wiki/Fork_(operating_system)
If you want something available from within Lua, you could try something like this:
How do you construct a read-write pipe with lua?
Lets say I am running a script and the game client waits for the script to be finished before it updates. Can Lua do somthing of a 'timeout'? Like, can i set a priority on the update so it leaves the script to do the update and then after words could it go back to where it was in the script?
You can also set a count hook with a suitable count for timeout and abort execution of the script in the hook.
Lua uses collaborative multithreading, so the script must know how long it has taken before it passes control back to the caller. Its not hard to figure out how long it has run using os.time and getting the difference. In some cases this might be more difficult, but if the script is a loop it shouldn't be hard. Once you've figured out that you've run for too long, do a coroutine.yield() and when you want to resume the script, simply call lua_resume from your update loop.
You can run your whole lua_State and lua script in another thread. When the Lua script accesses functions you implemented which need to modify things in the main thread, use mutexes and other things to make that access thread-safe.
This way, you can easily have your Lua script hang or do whatever while your main thread can continue to operate normally, however, it also requires you to make all your implemented functions accessing anything the main thread probably takes care of normally (like graphics) to be threading-aware.