I am new to Lua and want to ask whether it is possible to restrict lua syntax in config file? I know that config loading have to be performed in jail, but how we can cope with while 1 do end in config file we want to load? Is there a way to allow only strings, assignments and tables in config and if not, then what is the best way to check that lua file doesn't contain undesirable constructs? Is manual pre-parsing the only solution?
You seem to already know about "sandboxing" in Lua. So what's left is as you say malicious constructs like infinite loops. And to solve that you need to solve the Halting Problem. Which is not practical.
Instead of "manually" parsing and hoping you find all the malicious content (you won't), how about just running your Lua interpreter with a timer set so that the script will be interrupted if it takes longer than N seconds?
If you want to explicitly forbid certain constructs in Lua, you have to actually scan the file yourself. Note that there are valid uses for those constructs, even in config files, so you are restricting what the user can do.
It wouldn't be too hard to write a simple Lua lexer that ignores the contents of strings and comments, but errors on any of the Lua keywords other than return. Given proper sandboxing (ie: no functions are available to be called), that should be sufficient to weed out anything malicious.
Also, note that Lua 5.1 doesn't make it easy to keep the parser from parsing non-text data (ie: compiled Lua bytecode). 5.2 offers specific API support for forcing the loader to only recognize text and therefore reject bytecode.
Related
I'm doing a project which involves parsing the histories of common lisp repos. I need to parse them into list-of-lists or something like that. Ideally, I'd like to preserve as much of the original source file syntax as possible, in some way. For example, in the case of the text #+sbcl <something>, which I think means "If our current lisp is sbcl, read <something>, otherwise skip it", I'd like to get something like (#+ 'sbcl <something>).
I originally wrote a LALR parser in Python, which sort of worked, but it's not ideal for many reasons. I'm having a lot of difficulty getting correct output, and I have tons of special cases to add.
I figured that what I should really do is is use lisp itself, since it already has a lisp parser built in. If I could just read a file into sexps, I could dump it into something (cl-json would do) for further processing down the line.
Unfortunately, when I attempt to read https://github.com/fukamachi/woo/blob/master/src/woo.lisp, I get the error
There is no package with the name WOO.EV.TCP
which is of course coming from line 80 of that file, since that package is defined in src/ev/tcp.lisp, and we haven't read it.
Basically, is it possible to just read the file into sexps without caring whether the packages are defined or if they contain the relevant symbols? If so, how? I've tried looking at the hyperspec reader documentation, but I don't see anything that sounds relevant.
I'm out of practice with actually writing common lisp, but it seems potentially possible to hack around this by handling the undefined package condition by creating a blank package with that name, and handling the no-symbol-of-that-name-in-package condition by just interning a given symbol. I think. I don't know how to actually do this, I don't know if it would work, I don't know how many special cases would be involved. Offhand, the first condition is called no-such-package, but the second one (at least in sbcl) is called simple-error, so I don't even know how to determine whether this particular simple-error is the no-such-symbol-in-that-package error, let alone how to extract the relevant names from the condition, fix it, and restart. I'd really like to hear from a common lisp expert that this is the right thing to do here before I go down the road of trying to do it this way, because it will involve a lot of learning.
It also occurs to me that I could fix this by just sed-ing the file before reading it. E.g. turning woo.ev.tcp:start-listening-socket into, say, woo.ev.tcp===start-listening-socket. I don't particularly like this solution, and it's not clear that I wouldn't run into tons more ugly special cases, but it might work if there's no better answer.
I am almost sure there is no easy portable way to do this for a number of reasons.
(Just limiting things to the non-existent-package problem for now.)
First of all there is no portable access into the bit of the reader which decides that tokens are going to be symbols and then looks for package markers &c: that just happens according to the rules in 2.3. So you can't easily intervene in this.
Secondly there's not portably enough information in any kind of condition the reader might signal to be able to handle them.
There are several possible ways out of this bit of the problem.
If you felt sufficiently heroic you might be able to teach the reader that all of the token-starting characters are in fact things you control and then write a token-reader that somehow deals with the whole package thing by returning some object which isn't a symbol. But to do that you need to deal with numbers, and if you think that's simple, well, it's not.
If you felt less heroic you could write a more primitive token-reader which just doesn't even try to deal with anything except grabbing all the characters needed and returns some kind of object which wraps a string. This would avoid the whole number problem at the cost of losing a lot of intofmration.
If you don't care about portability, find an implementation, understand how its reader does it, and muck around with it. There are more open source or source-available implementations than I can easily count (perhaps I am not very good at counting) so this is a pretty good approach. It's certainly what I'd do.
But this is only the start of the problems. The CL reader is hairy and, in its standard configuration (the configuration which is used for things like compile-file unless people have arranged otherwise) can run completely arbitrary code at read time, including code which modifies the reader itself, some of which may do so in an implementation-dependent way. And people use this: there's a reason Lisp is called the 'programmable programming language' and it's that people program it.
I've decided to solve this using sed (actually Python's re.sub, but who's counting?) because it'll work for my actual use case, and was easy.
For future readers: The various people saying this is impossible in general are probably right. The other questions posted by #Svante look like good easy ways to solve part of the problem. Other parts of the problem might be solved more elegantly by replacing the reader macros for #., #+, #-, etc with ones which just make a list, which sounds less heroic than the suggestions from #tfb, but I don't have time for that shit.
I want to write a GUI frontend to gdb, using MI. Currently I can communicate with gdb via pipe, but a GUI debugger should be able to display source code and allow users to check/modify data using thier mouse.
The question is, in order to know what variable the user is pointing at, I think I need to write a parser. However, I don't want to implement the whole lexer and parser things. How can I get the locations of those identifiers in the source code?
[EDIT]
In short, I want the user to be able to check the value of a variable by hover over the variable using mouse, so I have to parse the code to know where does each variable appear. I want to achieve functions like this:
How can I get the locations of those identifiers in the source code?
... without writing a parser.
You can't. You would need to either write your own (for all programming languages your GUI will support), or hook one of the existing ones.
Clang makes it relatively easy to incorporate the C/C++ parser into a GUI, but ...
not everything can be parsed with Clang
this one aspect of writing a GUI is likely to be 100x more complicated than all the others, so perhaps not worth the effort.
I've seen answers to this question but I couldn't figure out which of the answers would perform the fastest. These are the answers I've seen- which is best?
Read one line at a time using each or each_line
Read one line at a time using gets
Save it all into an array of lines using readlines and then use each
Use grep (not sure what exactly to do with grep...)
Use sed (not sure what exactly to do with sed...)
Something else?
Also, would it be better to just use another language or should Ruby be fine?
EDIT:
More details: Each line contains something like "id1 attr1_1 attr2_1 id2 attr1_2 attr2_2... idn attr1_n attr2_n" (n is very big) and I need to insert those into a database. For that example line, I would need to insert n rows into the database.
Ruby will likely be using the same or very similar low-level code (written in C) to do the actual reading from disk for the first three options, so they should perform similarly. Given that, you should choose whichever is most convenient for you; the ability to do that is what makes languages like Ruby so useful! You will be reading a lot of data from disk, so I would suggest using each_line and processing each line as you read it.
I would not recommend bringing grep, sed, or any other such external utilities into the picture unless you have a very good reason, as they will make your code less portable and expose you to failures that may be difficult to diagnose.
If you're using Ruby then there's no need to worry about performance. The language is such that it suits an iterative approach to reading a file, line by line, and works very nicely. So long as you're using the language the way it's designed you can let the interpreter people worry about performance. Job done.
If one particular readLargeFileFast method is needed then it should be because it's really hindering the program somehow. Now, you write a C program to do it and popen it as a separate process within your ruby code. You could call it read_large.c and (perhaps) use command line arguments to tell it how to behave.
This is championing the idea that a scripting language is used for a fast development rather than a fast run time. As such a developer can be very productive by swiftly 'prototyping' a program in something like Ruby and only later rewriting the components warrant some low level code. Often, however, once it's working in script, it's not necessary to do anything else at all.
The Ruby Docs describe launching a separate process and treating it as a file. It's easy-peasy! A good start is The Art of Linux Programming's introductory paragraph on program modularity. This book also makes a great example of using linux's standard stream editor, called sed, which you could probably use from Ruby right now.
If you need to parse or edit a lot of text then many interpreters or editors have been written around sed's functionality. Further, it may save you a lot of effort writing something super efficient if you don't know C. Good is the Introduction to SED by Bruce Barnett.
Many games these days make available some lua scripting, but this is universally undocumented.
So let's say I can get a game to run my lua script (it's lua 5.1) - and the script can write what it finds to text files on disk. How much can I discover about environment the script is executing in?
For example it seems I can list keys in tables, and find out what's a function and what's some other type of object, but there's no obvious way to guess how many arguments function takes (and a mistake usually results in crash to desktop).
Most languages provide some reflection functionality that could be used here - how much is possible in embedded lua environment?
"debug" standard library has some functions, which you may find useful:
debug.getfenv - Returns the environment of object.
debug.getinfo - Returns a table with information about a function.
... and more
Lua Reference Manual also states:
several of these functions violate some assumptions about Lua code (e.g., that variables local to a function cannot be accessed from outside or that userdata metatables cannot be changed by Lua code) and therefore can compromise otherwise secure code.
So with debug library, you may access more.
Unfortunately, there is not much you can learn about functions in Lua - they by design accept any number of parameters. Without the ability to look at the sources, your only resort is the documentation and/or other samples.
The most you can do in this case is traverse the entire _G table recursively and dump every table/function, printing the results to a file.
"A mistake usually results in crash to desktop" is a sign of a really bad design - good API should tell you, that it expects A, and you passed B. For example in Lqt, a Qt binding to Lua, we check every parameter against the original Qt API, so that the programmer is notified of mistakes:
> QApplication.setFont(1, 2)
QApplication::setFont(number, number): incorrect or extra arguments, expecting: QFont*,string,.
Out of curiosity, I wonder what can people do with parsers, how they are applied, and what do people usually create with it?
I know it's widely used in programming language industry, however I think this is just a tiny portion of it, right?
Besides special-purpose languages, my most ambitious use of a parser generator yet (with good old yacc back in C, and again later with pyparsing in Python) was to extract, validate and possibly alter certain meta-info from SQL queries -- parsing SQL properly is a real challenge (especially if you hope to support more than one dialect!-), a parser generator (and a lexer it sits on top of) at least remove THAT part of the job!-)
They are used to parse text....
To give a more concrete example, where I work we use lexx/yacc to parse strings coming over sockets.
Also from the name it should give you an idea what javacc is used for (java compiler compiler!)
Generally to parse Domain Specific Languages or scripting languages, or similar support for code snipits.
Previously I have seen it used to parse the command line based output of another software tool. This way the outer tool (VPN software) could re-use the base router IPSec code without modification. As lots of what was being parsed was IP Route tables and other structured repeated text.
Using a parser allowed simple changes when the formatting changed, instead of trying to find and tweak the a hand written parser. And the output did change a few times of the life of the product.
I used parsers to help process +/- 800 Clipper source files into similar PRGs that could be compiled with Alaksa Xbase 32.
You can use it to extend your favorite language by getting its language definition from their repository and then adding what you've always wanted to have. You can pass the regular syntax to your application and handle the extension in your own program.