I'm trying to parse a syntax using the Shunting Yard (SY) algorithm. The syntax includes the following commands (they're are many many others though!)
a + b // a and b are numbers
setxy c d //c,d can be numbers
setxy c+d b+a //all numbers
Essentially, setxy is a function but it doesn't expect any function argument separators. This makes it very difficult (impossible?) to do via SY due to the lack of parens and function argument separators.
Any idea if SY can be used to parse a parentheses-less/function argument separator-less function or should I move on to a different parsing algorithm? If so, which one would you recommend?
Thanks!
djs22
Having defined correct grammar you can make http://www.antlr.org/ generate parser for you. Whether it is appropriate solution depends on your homework "requirements".
At least you can generate it and look inside for some hints.
I don't fully understand what you are trying to do, but perhaps you could use some regex? what are you trying to do write a simple command line program?
Related
I'm writing a tool which reads go code and parses some duration expressions like: dur := 5 * time.Minute. I already have the parsing step done and got a *ast.BinaryExpr. How can I evaluate this expression and get its value?
Is there something in the toolchain/packages or do I need to go by hand?
I think parser package is the one your looking for.
go also has a package named eval (https://godoc.org/github.com/apaxa-go/eval) that evaluates expressions.
And there are two libraries that might help you down the line.
https://github.com/PaesslerAG/gval
https://github.com/Knetic/govaluate
Gval (Go eVALuate) provides support for evaluating arbitrary expressions, in particular Go-like expressions.
Good luck!
I would write a function which can parse the multiplication of 2 algebraic expressions in GF(2), i.e any variable in the expression only take on 2 possible values 0 or 1, so a^2 = a,(0^2 = 0, 1^2 = 1)
As an example, if we expand (a+b)*(a+c) in GF(2), we should get
(a + b)*(a + c) = a^2 + a*b + a*c + b*c = a + a*b + a*c + b*c.
However, I am not sure how to start about the parsing of 2 algebraic expressions using strings. Any suggestion/ help is appreciated. Thanks!
I would recommend taking a look at OMeta, by Alex Warth, and/or PetitParser, by Lucas Rengli. Both are excellent frameworks for writing parsers. The first one is for JS, the second for Smalltalk.
Here are some few initial lines of code showing how to write your parser in PetitParser. Every fragment is a method of your own subclass of PPCompositeParser.
constant
ˆ$0 asParser / $1 asParser
variable
^#letter asParser
timesOp
^#blank asParser star , $* asParser, #blank asParser star
sumOp
^#blank asParser star, $* asParser, #blank asParser star
element
^self constant / self variable
term
^self element , (self timesOp , self element) star
etc.
I'm not saying this is trivial. I'm only saying that this is where I would start. Note also that once you have your grammar in place you might want to subclass it so you can generate more appropriate productions, etc.
Writing parsers for big complicated languages can be hard. But writing parsers for algebraic expressions (GF(2) or otherwise) is pretty easy.
See my SO answer on how to write such parsers easily: Is there an alternative for flex/bison that is usable on 8-bit embedded systems?
The GF(2) bit is about semantic interpretation of what such a formula means. It doesn't matter at all for parsing, which is purely about syntax.
Where meaning comes into play is when you want to interpret the formula.
At some point, you may want to evaluate the expression using values for the variables. To do that, you have to capture the formula as a data structure (usually called an (abstract) syntax tree), and then walk that tree to compute the desired result. That link also discusses how to do that.
If you want to manipulate the formula symbolically, you're in an entirely different ball game. Parsing is still easy, but formula manipulation is not, and you'll want to use tools that are designed to do such symbolic manipulation; they generally define thier own parsing machinery (and make it easy to use) to ensure that the captured parse can be manipulated. And of course, you'll have to define what the rules of you symbolic manipulation are.
You can see an example of how to write something pretty close to your needs at Symbolic Algebra with a program transformation system. (This a tool that my company builds).
I'm currently trying to parse math expression into expression tree.
But I'm stuck on the stage where I need to implement functions and negates. I don't understand logic to do it using Shunting-Yard algorithm.
What I currently want to do is to support
Negates, like -(x+5)
Function calls, like min(x,y)
Power just after function name, like cos^2(x)
Implicit multiplication, like 2x is same as 2*x
Scientific notation
Constants e and pi
Can somebody tell me hints how to implement this?
An working, PSR-0 compatible implementation of the shunting yard algorithm can be found here: https://github.com/andig/php-shunting-yard/tree/dev.
It supports constants, custom functions etc.
I'm making my own javascript-based programming language (yeah, it is crazy, but it's for learn only... maybe?). Well, I'm reading about parsers and the first pass is to convert the code source to tokens, like:
if(x > 5)
return true;
Tokenizer to:
T_IF "if"
T_LPAREN "("
T_IDENTIFIER "x"
T_GT ">"
T_NUMBER "5"
T_RPAREN ")"
T_IDENTIFIER "return"
T_TRUE "true"
T_TERMINATOR ";"
I don't know if my logic is correct for that for while. On my parser it is even better (or not?) and translate to it (yeah, multidimensional array):
T_IF "if"
T_EXPRESSION ...
T_IDENTIFIER "x"
T_GT ">"
T_NUMBER "5"
T_CLOSURE ...
T_IDENTIFIER "return"
T_TRUE "true"
I have some doubts:
Is my way better or worse that the original way? Note that my code will be read and compiled (translated to another language, like PHP), instead of interpreted all the time.
After I tokenizer, what I need do exactly? I'm really lost on this pass!
There are some good tutorial to learn how I can do it?
Well, is that. Bye!
Generally, you want to separate the functions of the tokeniser (also called a lexer) from other stages of your compiler or interpreter. The reason for this is basic modularity: each pass consumes one kind of thing (e.g., characters) and produces another one (e.g., tokens).
So you’ve converted your characters to tokens. Now you want to convert your flat list of tokens to meaningful nested expressions, and this is what is conventionally called parsing. For a JavaScript-like language, you should look into recursive descent parsing. For parsing expressions with infix operators of different precedence levels, Pratt parsing is very useful, and you can fall back on ordinary recursive descent parsing for special cases.
Just to give you a more concrete example based on your case, I’ll assume you can write two functions: accept(token) and expect(token), which test the next token in the stream you’ve created. You’ll make a function for each type of statement or expression in the grammar of your language. Here’s Pythonish pseudocode for a statement() function, for instance:
def statement():
if accept("if"):
x = expression()
y = statement()
return IfStatement(x, y)
elif accept("return"):
x = expression()
return ReturnStatement(x)
elif accept("{")
xs = []
while True:
xs.append(statement())
if not accept(";"):
break
expect("}")
return Block(xs)
else:
error("Invalid statement!")
This gives you what’s called an abstract syntax tree (AST) of your program, which you can then manipulate (optimisation and analysis), output (compilation), or run (interpretation).
Most toolkits split the complete process into two separate parts
lexer (aka. tokenizer)
parser (aka. grammar)
The tokenizer will split the input data into tokens. The parser will only operate on the token "stream" and build the structure.
Your question seems to be focused on the tokenizer. But your second solution mixes the grammar parser and the tokenizer into one step. Theoretically this is also possible but for a beginner it is much easier to do it the same way as most other tools/framework: keep the steps separate.
To your first solution: I would tokenize your example like this:
T_KEYWORD_IF "if"
T_LPAREN "("
T_IDENTIFIER "x"
T_GT ">"
T_LITARAL "5"
T_RPAREN ")"
T_KEYWORD_RET "return"
T_KEYWORD_TRUE "true"
T_TERMINATOR ";"
In most languages keywords cannot be used as method names, variable names and so on. This is reflected already on the tokenizer level (T_KEYWORD_IF, T_KEYWORD_RET, T_KEYWORD_TRUE).
The next level would take this stream and - by applying a formal grammar - would build some datastructure (often called AST - Abstract Syntax Tree) which might look like this:
IfStatement:
Expression:
BinaryOperator:
Operator: T_GT
LeftOperand:
IdentifierExpression:
"x"
RightOperand:
LiteralExpression
5
IfBlock
ReturnStatement
ReturnExpression
LiteralExpression
"true"
ElseBlock (empty)
Implementing the parser by hand is usually done by some frameworks. Implementing something like that by hand and efficiently is usually done at a university in the better part of a semester. So you really should use some kind of framework.
The input for a grammar parser framework is usually a formal grammar in some kind of BNF. Your "if" part migh look like this:
IfStatement: T_KEYWORD_IF T_LPAREN Expression T_RPAREN Statement ;
Expression: LiteralExpression | BinaryExpression | IdentifierExpression | ... ;
BinaryExpression: LeftOperand BinaryOperator RightOperand;
....
That's only to get the idea. Parsing a realworld-language like Javascript correctly is not an easy task. But funny.
Is my way better or worse that the original way? Note that my code will be read and compiled (translated to another language, like PHP), instead of interpreted all the time.
What's the original way ? There are many different ways to implement languages. I think yours is fine actually, I once tried to build a language myself that translated to C#, the hack programming language. Many language compilers translate to an intermediate language, it's quite common.
After I tokenizer, what I need do exactly? I'm really lost on this pass!
After tokenizing, you need to parse it. Use some good lexer / parser framework, such as the Boost.Spirit, or Coco, or whatever. There are hundreds of them. Or you can implement your own lexer, but that takes time and resources. There are many ways to parse code, I generally rely on recursive descent parsing.
Next you need to do Code Generation. That's the most difficult part in my opinion. There are tools for that too, but you can do it manually if you want to, I tried to do it in my project, but it was pretty basic and buggy, there's some helpful code here and here.
There are some good tutorial to learn how I can do it?
As I suggested earlier, use tools to do it. There are a lot of pretty good well-documented parser frameworks. For further information, you can try asking some people who know about this stuff. #DeadMG , over at the Lounge C++ is building a programming language called "Wide". You may try consulting him.
Let's say I have this statement in a programming language:
if (0 < 1) then
print("Hello")
The lexer will translate it into:
keyword: if
num: 0
op: <
num: 1
keyword: then
keyword: print
string: "Hello"
The parser will then take the information (aka "Token Stream") and make this:
if:
expression:
<:
0, 1
then:
print:
"Hello"
I don't know if this will help or not, but I hope it does.
I need to convert a math formula written in the Latex style to the function of a C/C++ code.
For example:
y = sin(x)^2 would become something like
double y = sin(x) * sin(x);
or
double y = pow(sin(x), 2);
where x is a variable defined somewhere before.
I mean that it should convert the latex formula to the C/C++ syntax. So that if there is a function y = G(x, y)^F(x) it doesn't matter what is G(x,y) and F(x),
it is a problem of the programmer to define it. It will just generate
double y = pow(G(x, y), F(x));
When the formula is too complicated it will take some time to make include it in the C/C++ formula. Is there any way to do this conversion?
Emacs' built-in calculator calc-mode can do this (and much more). Your examples can be converted like this:
Put the formula in some emacs buffer
$ y = sin(x)^2 $
With the cursor in the formula, activate calc-embedded mode
M-x calc-embedded
Switch the display language to C:
M-x calc-c-language
There you are:
$ y == pow(sin(x), 2) $
Note that it interprets the '=' sign in latex as an equality, which results in '==' for C. The latex equivalent to Cs assignment operator '=' would be '\gets'.
More on this topic on Turong's blog
I know the question is too old, but I'll just add a reply anyway as a think it might help someone else later. The question popped up a lot for me in my searches.
I'm working on a tool that does something similar, in a public git repo
You'll have to put some artificial limitations on your latex input, that's out of question.
Currently the tool I wrote only supports mul, div, add, sub, sqrt, pow, frac and sum as those are the only set of operations I need to handle, and the imposed limitations can be a bit loose by providing a preprocessor (see preproc.l for an [maybe not-so-good] example) that would clean away the raw latex input.
A mathematical equation, such as the ones in LaTeX, and a C expression are not interchangeable. The former states a relation between two terms, the latter defines an entity that can be evaluated, unambiguously yielding one value. a = b in C means 'take the value in variable b and store it in variable a', wheres in Math, it means 'in the current context, a and b are equal'. The first describes a computation process, the second describes a static fact. Consequently, the Math equation can be reversed: a = b is equivalent to b = a, but doing the same to the C equation yields something quite different.
To make matters worse, LaTeX formulae only contain the information needed to render the equations; often, this is not enough to capture their meaning.
Of course some LaTeX formulae, like your example, can be converted into C computations, but many others cannot, so any automated way of doing so would only make limited sense.
I'm not sure there is a simple answer, because mathematical formulaes (in LaTeX documents) are actually ambiguous, so to automate their translation to some code requires automating their understanding.
And the MathML standard has, IIRC, two forms representing formulaes (one for displaying, another for computing) and there is some reason for that.