I am trying to write an interpreter but am having difficulty understanding the theoretical underpinnings of the process.
I understand that the first part is to write a lexer which splits the string up into a list of valid tokens and then a parser is used to generate the corresponding abstract syntax tree for this string of tokens. However, a parser is built using a grammar rule, which is what I'm having difficulty understanding.
A Grammar rule is obviously used to create the rules of the resulting abstract syntax tree, but how exactly does this middle step work. Does it pattern match on string characters and a specific list of tokens or . . .?
Any type of intuition or explanation is welcomed. Thanks!
Search the internet for lex/yacc examples and tutorials. Learning by doing.
Ability to program in C is also necessary.
http://ds9a.nl/lex-yacc/cvs/lex-yacc-howto.html
lex is the ancient Unix lexer, which generates C-code from a regexp based spec.
yacc is the ancient Unix parser for building syntax trees. It generates C code too.
The modern GNU versions of the tools are called flex and bison.
Here's the core of the yacc code of a calculator. It shows how higher level constructs are built from the tokens, and what to do when such constructs are encountered.
%%
list : // empty
| list stm '\n' { print(); }
| list cmd '\n' { print(); }
| list cmd stm '\n' { print(); }
| list stm cmd '\n' { print(); }
| list cmd stm cmd '\n' { print(); }
| list error '\n' { yyerrok; print(); }
;
cmd : COMMAND { commands[$1](); }
;
stm : expr { output = $1; outputPush(); }
| VAR '=' expr { vars_set($1, &$3); }
;
expr : { outputGet(); $$ = output; }
| '_' { outputGet(); $$ = output; }
| '(' expr ')' { $$ = $2; }
| expr OPADD expr { $$ = tNumOpIn ($1, $2, $3); }
| expr OPMUL expr { $$ = tNumOpIn ($1, $2, $3); }
| expr OPPOW expr { $$ = tNumOpIn ($1, $2, $3); }
| OPPRE expr { $$ = tNumOpPre($1, $2); }
| VAR { if (vars_get($1,&$$)) $$=output; }
| NUMBER { $$ = $1; }
;
%%
Related
I have a grammar for arithmetic expression which solves number of expression (one per line) in a text file. While compiling YACC I am getting message 2 shift reduce conflicts. But my calculations are proper. If parser is giving proper output how does it resolves the shift/reduce conflict. And In my case is there any way to solve it in YACC Grammar.
YACC GRAMMAR
Calc : Expr {printf(" = %d\n",$1);}
| Calc Expr {printf(" = %d\n",$2);}
| error {yyerror("\nBad Expression\n ");}
;
Expr : Term { $$ = $1; }
| Expr '+' Term { $$ = $1 + $3; }
| Expr '-' Term { $$ = $1 - $3; }
;
Term : Fact { $$ = $1; }
| Term '*' Fact { $$ = $1 * $3; }
| Term '/' Fact { if($3==0){
yyerror("Divide by Zero Encountered.");
break;}
else
$$ = $1 / $3;
}
;
Fact : Prim { $$ = $1; }
| '-' Prim { $$ = -$2; }
;
Prim : '(' Expr ')' { $$ = $2; }
| Id { $$ = $1; }
;
Id :NUM { $$ = yylval; }
;
What change should I do to remove such conflicts in my grammar ?
Bison/yacc resolves shift-reduce conflicts by choosing to shift. This is explained in the bison manual in the section on Shift-Reduce conflicts.
Your problem is that your input is just a series of Exprs, run together without any delimiter between them. That means that:
4 - 2
could be one expression (4-2) or it could be two expressions (4, -2). Since bison-generated parsers always prefer to shift, the parser will choose to parse it as one expression, even if it were typed on two lines:
4
-2
If you want to allow users to type their expressions like that, without any separator, then you could either live with the conflict (since it is relatively benign) or you could codify it into your grammar, but that's quite a bit more work. To put it into the grammar, you need to define two different types of Expr: one (which is the one you use at the top level) cannot start with an unary minus, and the other one (which you can use anywhere else) is allowed to start with a unary minus.
I suspect that what you really want to do is use newlines or some other kind of expression separator. That's as simple as passing the newline through to your parser and changing Calc to Calc: | Calc '\n' | Calc Expr '\n'.
I'm sure that this appears somewhere else on SO, but I can't find it. So here is how you disallow the use of unary minus at the beginning of an expression, so that you can run expressions together without delimiters. The non-terminals starting n_ cannot start with a unary minus:
input: %empty | input n_expr { /* print $2 */ }
expr: term | expr '+' term | expr '-' term
n_expr: n_term | n_expr '+' term | n_expr '-' term
term: factor | term '*' factor | term '/' factor
n_term: value | n_term '+' factor | n_term '/' factor
factor: value | '-' factor
value: NUM | '(' expr ')'
That parses the same language as your grammar, but without generating the shift-reduce conflict. Since it parses the same language, the input
4
-2
will still be parsed as a single expression; to get the expected result you would need to type
4
(-2)
Let's assume I have grammar like this:
expr : expr '-' expr { $$ = $1 - $3; }
| "Function" '(' expr ',' expr ')' { $$ = ($3 - $5) * 2; }
| NUMBER { $$ = $1; };
How can use rule
expr : expr '-' expr { $$ = $1 - $3; }
inside
expr : "Function" '(' expr ',' expr ')' { $$ = ($3 - $5) * 2; }
Because implementation of $1 - $3 is repeated? It would be much better if I can use already implemented subtraction from rule one and only add multiplication with 2. This is just the basic example, but I have very big grammar with lot of repeating calculations.
I have the following parser grammar (this is a small sample):
expr:
ident assignop expr
{
$$ = new NAssignment(new NAssignmentIdentifier(*$1), $2, *$3);
} |
STAR expr %prec IDEREF
{
$$ = new NDereferenceOperator(*$2);
} |
STAR expr assignop expr %prec IDEREF
{
$$ = new NAssignment(new NAssignmentDereference(*$2), $3, *$4);
} |
... ;
...
assignop:
ASSIGN_EQUAL |
ASSIGN_ADD |
ASSIGN_SUBTRACT |
ASSIGN_MULTIPLY |
ASSIGN_DIVIDE ;
Now I'm trying to parse any of the following lines:
*0x8000 = 0x7000;
*mem = 0x7000;
However, Bison keeps seeing "*mem" and reducing on the 'STAR expr' rule and not performing look-ahead to see whether 'STAR expr assignop...' matches. As far as I understand Bison, it should be doing this look-ahead. My closest guess is that %prec is turning off look-ahead or something strange like that, but I can't see why it would do so (since the prec values are equivalent).
How do I make it perform look-ahead in this case?
EDIT:
The state that it enters when encountering 'STAR expr' is:
state 45
28 expr: STAR expr .
29 | STAR expr . assignop expr
35 | expr . binaryop expr
$default reduce using rule 28 (expr)
assignop go to state 81
binaryop go to state 79
So I don't understand why it's picking $default when it could pick assignop (note that the order of the rules in the parser.y file don't affect which one it picks in this case; I've tried reordering the assignop one above the standard 'STAR expr').
This will happen if IDREF is higher precedence than ASSIGN_EQUAL, ASSIGN_ADD, etc. Specifically, in this case with the raw parser (before precedence is applies), you have shift/reduce conflicts between the expr: STAR expr rule and the various ASSIGN_XXX tokens. The precedence rules you have resolve all the conflicts in favor of the reduce.
The assignop in the state is a goto, not a shift or reduce, so doesn't enter into the lookahead or token handling at all -- gotos only occur after some token has been shifted and then later reduced to the non-terminal in question.
I ended up solving this problem by creating another rule 'deref' like so:
deref:
STAR ident
{
$$ = new NDereferenceOperator(*$<ident>2);
} |
STAR numeric
{
$$ = new NDereferenceOperator(*$2);
} |
STAR CURVED_OPEN expr CURVED_CLOSE
{
$$ = new NDereferenceOperator(*$3);
} |
deref assignop expr
{
if ($1->cType == "expression-dereference") // We can't accept NAssignments as the deref in this case.
$$ = new NAssignment(new NAssignmentDereference(((NDereferenceOperator*)$1)->expr), $2, *$3);
else
throw new CompilerException("Unable to apply dereferencing assignment operation to non-dereference operator based LHS.");
} ;
replacing both rules in 'expr' with a single 'deref'.
I tried to extend the example grammar that comes as part of the "F# Parsed Language Starter" to support unary minus (for expressions like 2 * -5).
I hit a block like Samsdram here
Basically, I extended the header of the .fsy file to include precedence like so:
......
%nonassoc UMINUS
....
and then the rules of the grammar like so:
...
Expr:
| MINUS Expr %prec UMINUS { Negative ($2) }
...
also, the definition of the AST:
...
and Expr =
| Negative of Expr
.....
but still get a parser error when trying to parse the expression mentioned above.
Any ideas what's missing? I read the source code of the F# compiler and it is not clear how they solve this, seems quite similar
EDIT
The precedences are ordered this way:
%left ASSIGN
%left AND OR
%left EQ NOTEQ LT LTE GTE GT
%left PLUS MINUS
%left ASTER SLASH
%nonassoc UMINUS
Had a play around and managed to get the precedence working without the need for %prec. Modified the starter a little though (more meaningful names)
Prog:
| Expression EOF { $1 }
Expression:
| Additive { $1 }
Additive:
| Multiplicative { $1 }
| Additive PLUS Multiplicative { Plus($1, $3) }
| Additive MINUS Multiplicative { Minus($1, $3) }
Multiplicative:
| Unary { $1 }
| Multiplicative ASTER Unary { Times($1, $3) }
| Multiplicative SLASH Unary { Divide($1, $3) }
Unary:
| Value { $1 }
| MINUS Value { Negative($2) }
Value:
| FLOAT { Value(Float($1)) }
| INT32 { Value(Integer($1)) }
| LPAREN Expression RPAREN { $2 }
I also grouped the expressions into a single variant, as I didn't like the way the starter done it. (was awkward to walk through it).
type Value =
| Float of Double
| Integer of Int32
| Expression of Expression
and Expression =
| Value of Value
| Negative of Expression
| Times of Expression * Expression
| Divide of Expression * Expression
| Plus of Expression * Expression
| Minus of Expression * Expression
and Equation =
| Equation of Expression
Taking code from my article Parsing text with Lex and Yacc (October 2007).
My precedences look like:
%left PLUS MINUS
%left TIMES DIVIDE
%nonassoc prec_uminus
%right POWER
%nonassoc FACTORIAL
and the yacc parsing code is:
expr:
| NUM { Num(float_of_string $1) }
| MINUS expr %prec prec_uminus { Neg $2 }
| expr FACTORIAL { Factorial $1 }
| expr PLUS expr { Add($1, $3) }
| expr MINUS expr { Sub($1, $3) }
| expr TIMES expr { Mul($1, $3) }
| expr DIVIDE expr { Div($1, $3) }
| expr POWER expr { Pow($1, $3) }
| OPEN expr CLOSE { $2 }
;
Looks equivalent. I don't suppose the problem is your use of UMINUS in capitals instead of prec_uminus in my case?
Another option is to split expr into several mutually-recursive parts, one for each precedence level.
I'm starting to play with Fslex/Fsyacc. When trying to generate the parser using this input
Parser.fsy:
%{
open Ast
%}
// The start token becomes a parser function in the compiled code:
%start start
// These are the terminal tokens of the grammar along with the types of
// the data carried by each token:
%token <System.Int32> INT
%token <System.String> STRING
%token <System.String> ID
%token PLUS MINUS ASTER SLASH LT LT EQ GTE GT
%token LPAREN RPAREN LCURLY RCURLY LBRACKET RBRACKET COMMA
%token ARRAY IF THEN ELSE WHILE FOR TO DO LET IN END OF BREAK NIL FUNCTION VAR TYPE IMPORT PRIMITIVE
%token EOF
// This is the type of the data produced by a successful reduction of the 'start'
// symbol:
%type <Ast.Program> start
%%
// These are the rules of the grammar along with the F# code of the
// actions executed as rules are reduced. In this case the actions
// produce data using F# data construction terms.
start: Prog { Program($1) }
Prog:
| Expr EOF { $1 }
Expr:
// literals
| NIL { Ast.Nil ($1) }
| INT { Ast.Integer($1) }
| STRING { Ast.Str($1) }
// arrays and records
| ID LBRACKET Expr RBRACKET OF Expr { Ast.Array ($1, $3, $6) }
| ID LCURLY AssignmentList RCURLY { Ast.Record ($1, $3) }
AssignmentList:
| Assignment { [$1] }
| Assignment COMMA AssignmentList {$1 :: $3 }
Assignment:
| ID EQ Expr { Ast.Assignment ($1,$3) }
Ast.fs
namespace Ast
open System
type Integer =
| Integer of Int32
and Str =
| Str of string
and Nil =
| None
and Id =
| Id of string
and Array =
| Array of Id * Expr * Expr
and Record =
| Record of Id * (Assignment list)
and Assignment =
| Assignment of Id * Expr
and Expr =
| Nil
| Integer
| Str
| Array
| Record
and Program =
| Program of Expr
Fsyacc reports the following error: "FSYACC: error FSY000: An item with the same key has already been added."
I believe the problem is in the production for AssignmentList, but can't find a way around...
Any tips will be appreciated
Hate answering my own questions, but the problem was here (line 15 of parser input file)
%token PLUS MINUS ASTER SLASH LT LT EQ GTE GT
Note the double definition (should have been LTE)
My vote goes for finding a way to improve the output of the Fslex/Fsyacc executables/msbuild tasks