Yacc parser for nested while loop - parsing

i was trying to code a parser using yacc and lex that count the number of nested loops (while or for).I started the implementation for just while loops.But for some reason the parser gives me an error at the end of a closing brace.
Here is the code.
%{
#include<stdio.h>
/*parser for counting while loops*/
extern int yyerror(char* error);
int while_count=0;
extern int yylex();
%}
%token NUMBER
%token VAR
%token WHILE
%%
statement_list : statement'\n'
| statement_list statement'\n'
;
statement :
while_stmt '\n''{' statement_list '}'
| VAR '=' NUMBER ';'
;
while_stmt :
WHILE '('condition')' {while_count++;}
;
condition :
VAR cond_op VAR
;
cond_op : '>'
| '<'
| '=''='
| '!''='
;
%%
int main(void){
yyparse();
printf("while count:%d\n",while_count);
}
int yyerror(char *s){
printf("Error:%s\n",s);
return 1;
}
what is wrong with that code.And is there a way in yacc to mention optional arguments? like the "\n" after while?
here is the lexer code
%{
#include"y.tab.h"
/*lexer for scanning nested while loops*/
%}
%%
[\t ] ; /*ignore white spaces*/
"while" {return WHILE;}
[a-zA-Z]+ {return VAR;}
[0-9]+ {return NUMBER;}
'$' {return 0;}
'\n' {return '\n' ;}
. {return yytext[0];}
%%
VAR is a variable name with just ascii characters and WHILE is the keyword while.type is not taken into consideration on variable assignments

The problem you seem to be having is with empty loop bodies, not nested loops. As written, your grammar requires at least one statement in the while loop body. You can fix this by allowing empty statement lists:
statement_list: /* empty */
| statement_list statement '\n'
;
You also ask about making newlines optional. The easiest way is to make the lexer simply discard newlines (as whitespace) rather than returning them. Then just get rid of the newlines in the grammar, and newlines can appear between any two tokens and will be ignored.
If you really must have newlines in the grammar for some reason, you can add a rule like:
opt_newlines: /* empty */ | opt_newlines '\n' ;
and then use this rule wherever you want to allow for newlines (replace all the literal '\n' in your grammar.) You have to be careful not to use it redundantly, however. If you do something like:
statement_list: /* empty */
| statement_list statement opt_newlines
;
while_stmt opt_newlines '{' opt_newlines statement_list opt_newlines '}'
you'll get shift/reduce conflicts as newlines before the } in a loop could be either part of the opt_newlines in the while or the opt_newlines in the statement_list. Its pretty easy to deal such conflicts by just removing the redundant opt_newlines.

Related

Bison - Productions for longest matching expression

I am using Bison, together with Flex, to try and parse a simple grammar that was provided to me. In this grammar (almost) everything is considered an expression and has some kind of value; there are no statements. What's more, the EBNF definition of the grammar comes with certain ambiguities:
expression OP expression where op may be '+', '-' '&' etc. This can easily be solved using bison's associativity operators and setting %left, %right and %nonassoc according to common language standards.
IF expression THEN expression [ELSE expression] as well as DO expression WHILE expression, for which ignoring the common case dangling else problem I want the following behavior:
In if-then-else as well as while expressions, the embedded expressions are taken to be as long as possible (allowed by the grammar). E.g 5 + if cond_expr then then_expr else 10 + 12 is equivalent to 5 + (if cond_expr then then_expr else (10 + 12)) and not 5 + (if cond_expr then then_expr else 10) + 12
Given that everything in the language is considered an expression, I cannot find a way to re-write the production rules in a form that does not cause conflicts. One thing I tried, drawing inspiration from the dangling else example in the bison manual was:
expression: long_expression
| short_expression
;
long_expression: short_expression
| IF long_expression THEN long_expression
| IF long_expression long_expression ELSE long_expression
| WHILE long_expression DO long_expression
;
short_expression: short_expression '+' short_expression
| short_expression '-' short_expression
...
;
However this does not seem to work and I cannot figure out how I could tweak it into working. Note that I (assume I) have resolved the dangling ELSE problem using nonassoc for ELSE and THEN and the above construct as suggested in some book, but I am not sure this is even valid in the case where there are not statements but only expressions. Note as well as that associativity has been set for all other operators such as +, - etc. Any solutions or hints or examples that resolve this?
----------- EDIT: MINIMAL EXAMPLE ---------------
I tried to include all productions with tokens that have specific associativity, including some extra productions to show a bit of the grammar. Notice that I did not actually use my idea explained above. Notice as well that I have included a single binary and unary operator just to make the code a bit shorter, the rules for all operators are of the same form. Bison with -Wall flag finds no conflicts with these declarations (but I am pretty sure they are not 100% correct).
%token<int> INT32 LET IF WHILE INTEGER OBJECTID TYPEID NEW
%right <str> THEN ELSE STR
%right '^' UMINUS NOT ISNULL ASSIGN DO IN
%left '+' '-'
%left '*' '/'
%left <str> AND '.'
%nonassoc '<' '='
%nonassoc <str> LOWEREQ
%type<ast_expr> expression
%type ...
exprlist: expression { ; }
| exprlist ';' expression { ; };
block: '{' exprlist '}' { ; };
args: %empty { ; }
| expression { ; }
| args ',' expression { ; };
expression: IF expression THEN expression { ; }
| IF expression THEN expression ELSE expression { ; }
| WHILE expression DO expression { ; }
| LET OBJECTID ':' type IN expression { ; }
| NOT expression { /* UNARY OPERATORS */ ; }
| expression '=' expression { /* BINARY OPERATORS */ ; }
| OBJECTID '(' args ')' { ; }
| expression '.' OBJECTID '(' args ')' { ; }
| NEW TYPEID { ; }
| STR { ; }
| INTEGER { ; }
| '(' ')' { ; }
| '(' expression ')' { ; }
| block { ; }
;
The following associativity declarations resolved all shift/reduce conflicts and produced the expected output (in all tests I could think of at least):
...
%right <str> THEN ELSE
%right DO IN
%right ASSIGN
%left <str> AND
%right NOT
%nonassoc '<' '=' LOWEREQ
%left '+' '-'
%left '*' '/'
%right UMINUS ISNULL
%right '^'
%left '.'
...
%%
...
expression: IF expression THEN expression
| IF expression THEN expression ELSE expression
| WHILE expression DO expression
| LET OBJECTID ':' type IN expression
| LET OBJECTID ':' type ASSIGN expression IN expression
| OBJECTID ASSIGN expression
...
| '-' expression %prec UMINUS
| expression '=' expression
...
| expression LOWEREQ expression
| OBJECTID '(' args ')'
...
...
Notice that the order of declaration of associativity and precedence rules for all symbols matters! I have not included all the production rules but if-else-then, while-do, let in, unary and binary operands are the ones that produced conflicts or wrong results with different associativity declarations.

antlr4: actions 'causing' left-recursion errors

The grammar below is failing to generate a parser with this error:
error(119): Sable.g4::: The following sets of rules are mutually left-recursive [expression]
1 error(s)
The error disappears when I comment out the embedded actions, but comes back when I make them effective.
Why are the actions bringing about such left-recursion that antlr4 cannot handle it?
grammar Sable;
options {}
#header {
package org.sable.parser;
import org.sable.parser.internal.*;
}
sourceFile : statement* EOF;
statement : expressionStatement (SEMICOLON | NEWLINE);
expressionStatement : expression;
expression:
{System.out.println("w");} expression '+' expression {System.out.println("tf?");}
| IDENTIFIER
;
SEMICOLON : ';';
RPARENTH : '(';
LPARENTH : ')';
NEWLINE:
('\u000D' '\u000A')
| '\u000A'
;
WS : WhiteSpaceNotNewline -> channel(HIDDEN);
IDENTIFIER:
(IdentifierHead IdentifierCharacter*)
| ('`'(IdentifierHead IdentifierCharacter*)'`')
;
fragment DecimalDigit :'0'..'9';
fragment IdentifierHead:
'a'..'z'
| 'A'..'Z'
;
fragment IdentifierCharacter:
DecimalDigit
| IdentifierHead
;
// Non-newline whitespaces are defined apart because they carry meaning in
// certain contexts, e.g. within space-aware operators.
fragment WhiteSpaceNotNewline : [\u0020\u000C\u0009u000B\u000C];
UPDATE: the following workaround kind of solves this specific situation, but not the case when init/after-like actions are needed on a lesser scope - per choice, not per rule.
expression
#init {
enable(Token.HIDDEN_CHANNEL);
}
#after {
disable(Token.HIDDEN_CHANNEL);
}:
expression '+' expression
| IDENTIFIER
;

parser for expressions in lex/yacc

I'm trying to parse statements to see whether they are valid or not according to these rules:
assignment:
id = exp ;
expression:
id op id {op id}
id is combination of digits and char, the first position contain a char.
I'm getting syntax error when I have something like this in my in.txt file: hellow = three3
but that shouldn't be a syntax error, and then when i put something like: hellow = =
that does not display a syntax error but it should. What am i doing wrong?
Lex:
%{
#include "y.tab.h"
#include <stdio.h>
%}
%%
[ \t\n]+ ;
[a-zA-Z][a-zA-Z0-9]* {
ECHO;
return ID;
}
%%
YACC:
%{
#include <stdio.h>
extern FILE * yyin;
%}
%token ID
%left '+' '-'
%left '*' '/' '%'
%right ';'
%%
assignment: expression
|
ID '=' expression
;
expression: ID
|
expression '*' expression
|
expression '/' expression
|
expression '%' expression
|
expression '+' expression
|
expression '-' expression
|
'(' expression ')'
;
%%
int main(void) {
yyin = fopen("in.txt", "r");
yyparse();
fclose(yyin);
return 0;
}
I can't say from just looking at this, but a good way to find out is running your parser when the environment variable YYDEBUG is 1, like in (unix):
YYDEBUG=1 ./a.out
You should get a detailed list of the steps the parser takes and the tokens it receives from the lexer (where the error is in most cases, like in your example: how does the parser ever get the '=' sign and the operators?). Compare this with the y.output file you get when you run yacc -v

can't find a simple error when parsing with YACC

i'm trying to make a very simple YACC parser on Pascal language which just includes integer declarations, some basic expressions and if-else statements. however, i cant find the error for hours and i'm going to be crazy soon. terminal says Error at line:0 but it is impossible!. i use flex and byacc for parser.i will be very glad if you can help me. this is my lex file as you can see;
%{
#include <stdio.h>
#include <string.h>
#include "y.tab.h"
extern int yylval;
int linenum=0;
%}
digit [0-9]
letter [A-Za-z]
%%
if return IF;
then return THEN;
else return ELSE;
for return FOR;
while return WHILE;
PROGRAM return PROGRAM_SYM;
BEGIN return BEGIN_SYM;
VAR return VAR_SYM;
END return END_SYM;
INTEGER return INTEGER_SYM;
{letter}({letter}|{digit})* return identifier;
[0-9]+ return NUMBER;
[\<][\=] return CON_LE;
[\>][\=] return CON_GE;
[\=] return CON_EQ;
[\:][\=] return ASSIGNOP;
; return semiColon;
, return comma;
\n {linenum++;}
. return (int) yytext[0];
%%
and this is my Yacc file
%{
#include <stdio.h>
#include <string.h>
#include "y.tab.h"
extern FILE *yyin;
extern int linenum;
%}
%token PROGRAM_SYM VAR_SYM BEGIN_SYM END_SYM INTEGER_SYM NUMBER
%token identifier INTEGER ASSIGNOP semiColon comma THEN
%token IF ELSE FOR WHILE
%token CON_EQ CON_LE CON_GE GE LE
%left '*' '/'
%left '+' '-'
%start program
%%
program: PROGRAM_SYM identifier semiColon VAR_SYM dec_block BEGIN_SYM statement_list END_SYM '.'
;
dec_block:
dec_list semiColon;
dec_list:
dec_list dec
|
dec
;
dec:
int_dec_list
;
int_dec_list:
int_dec_list int_dec ':' type
|
int_dec ':' type
;
int_dec:
int_dec comma identifier
|
identifier
;
type:
INTEGER_SYM
;
statement_list:
statement_list statement
|
statement
;
statement:
assignment_list
|
expression_list
|
selection_list
;
assignment_list:
assignment_list assignment
|
assignment
;
assignment:
identifier ASSIGNOP expression_list
;
expression_list:
expression_list expression semiColon
|
expression semiColon
;
expression:
'(' expression ')'
|
expression '*' expression
|
expression '/' expression
|
expression '+' expression
|
expression '-' expression
|
factor
;
factor:
identifier
|
NUMBER
;
selection_list:
selection_list selection
|
selection
;
selection:
IF '(' logical_expression ')' THEN statement_list ELSE statement_list
;
logical_expression:
logical_expression '=' expression
|
logical_expression '>' expression
|
logical_expression '<' expression
;
%%
void yyerror(char *s){
fprintf(stderr,"Error at line: %d\n",linenum);
}
int yywrap(){
return 1;
}
int main(int argc, char *argv[])
{
/* Call the lexer, then quit. */
yyin=fopen(argv[1],"r");
yyparse();
fclose(yyin);
return 0;
}
and finally i take an error at the first line when i give the input;
PROGRAM myprogram;
VAR
i:INTEGER;
i3:INTEGER;
j:INTEGER;
BEGIN
i := 3;
j := 5;
i3 := i+j*2;
i := j*20;
if(i>j)
then i3 := i+50+(45*i+(40*j));
else i3 := i+50+(45*i+(40*j))+i+50+(45*i+(30*j));
END.
For debugging grammars, YYDEBUG is your friend. Either stick #define YYDEBUG 1 in the %{..%} in the top of your .y file, or compile with -DYYDEBUG, and stick a yydebug = 1; in main before calling yyparse and you'll get a slew of info about what tokens the parser is seeing and what it is doing with them....
Your lexical analyzer returns blanks and tabs as tokens, but the grammar doesn't recognize them.
Add a parser rule:
[ \t\r] { }
This gets you to line 6 instead of line 0 before you run into an error. You get that error because you don't allow semicolons between declarations:
dec_block:
dec_list semiColon;
dec_list:
dec_list dec
|
dec
;
dec:
int_dec_list
;
That should probably be:
dec_block:
dec_block dec
|
dec
;
dec:
int_dec_list semiColon
;
Doing this gets you to line 14 in the input.
Incidentally, one of the first things I did was to ensure that the lexical analyzer tells me what it is doing, by modifying the rules like this:
if { printf("IF\n"); return IF; }
In long term code, I'd make that diagnostic output selectable at run-time.
You have a general problem with where you expect semicolons. It is also not clear that you should allow an expression_list in the rule for statement (or, maybe, 'not yet' — that might be appropriate when you have function calls, but allowing 3 + 2 / 4 as a 'statement' is not very helpful).
This grammar gets to the end of the input:
%{
#include <stdio.h>
#include <string.h>
#include "y.tab.h"
extern FILE *yyin;
extern int linenum;
%}
%token PROGRAM_SYM VAR_SYM BEGIN_SYM END_SYM INTEGER_SYM NUMBER
%token identifier INTEGER ASSIGNOP semiColon comma THEN
%token IF ELSE FOR WHILE
%token CON_EQ CON_LE CON_GE GE LE
%left '*' '/'
%left '+' '-'
%start program
%%
program: PROGRAM_SYM identifier semiColon VAR_SYM dec_block BEGIN_SYM statement_list END_SYM '.'
;
dec_block:
dec_block dec
|
dec
;
dec:
int_dec_list semiColon
;
int_dec_list:
int_dec_list int_dec ':' type
|
int_dec ':' type
;
int_dec:
int_dec comma identifier
|
identifier
;
type:
INTEGER_SYM
;
statement_list:
statement_list statement
|
statement
;
statement:
assignment
|
selection
;
assignment:
identifier ASSIGNOP expression semiColon
;
expression:
'(' expression ')'
|
expression '*' expression
|
expression '/' expression
|
expression '+' expression
|
expression '-' expression
|
factor
;
factor:
identifier
|
NUMBER
;
selection:
IF '(' logical_expression ')' THEN statement_list ELSE statement_list
;
logical_expression:
expression '=' expression
|
expression '>' expression
|
expression '<' expression
;
%%
void yyerror(char *s){
fprintf(stderr,"Error at line: %d\n",linenum);
}
int yywrap(){
return 1;
}
int main(int argc, char *argv[])
{
/* Call the lexer, then quit. */
yyin=fopen(argv[1],"r");
yyparse();
fclose(yyin);
return 0;
}
Key changes include removing assignment_list and expression_list, and modifying logical_expression so that the two sides of the expansion are expression, rather than the LHS being logical_expression (which then never had a primitive definition, leading to the problems with warnings).
There are still issues to resolve; the expression_list in the selection should be more restrictive to accurately reflect the grammar of Pascal. (You need a block where that could be a single statement or a BEGIN, statement list, END.)

ANTLR Grammar With Ambiguous Alternatives

My grammar is producing an unexpected result. I am not sure if it is just my bug or some issues with ANTLR'S ambiguous alternatives processing logic.
Here is my grammar :
grammar PPMacro;
options {
language=Java;
backtrack=true;
}
file: (inputLines)+ EOF;
inputLines
: ( preprocessorLineSet | oneNormalInputLine ) ;
oneNormalInputLine #after{System.out.print("["+$text+"]");}
: (any_token_except_crlf)* CRLF ;
preprocessorLineSet
: ifPart endifLine;
ifPart: ifLine inputLines* ;
ifLine #after{System.out.print("{"+$text+"}" );}
: '#' IF (any_token_except_crlf)* CRLF ;
endifLine #after{System.out.print("{"+$text+"}" );}
: '#' ENDIF (any_token_except_crlf)* CRLF ;
any_token_except_crlf: (ANY_ID | WS | '#'|IF|ENDIF);
// just matches everything
CRLF: '\r'? '\n' ;
WS: (' '|'\t'|'\f' )+;
Hash: '#' ;
IF : 'if' ;
ENDIF : 'endif' ;
ANY_ID: ( 'a'..'z'|'A'..'Z'|'0'..'9'| '_')+ ;
Explanation:
It is for parsing a C++ #if ... #endif block
I am trying to recognize nested #if #endif block. This is done by my preprocessorLineSet. It contains a recursive definition to support nested block. oneNormalInputLine is to handle anything not of the #if form. This rule is a match anything rule and actually matches a #if line. But I deliberately put it after the preprocessorLineSet in inputLines. I'm expecting this ordering can prevent it from matching a #if or #endif line. The reason to use a catch-all rule is that I want a rule to accept any other c++ syntax and simply echo them back to the output.
I my test, I just print out everything. Lines matched by preprocessorLineSet should be surrounded by {}, while those matched by oneNormalInputLine should be surrounded by [].
Sample inputs :
#if s
s
#if a
s
s
#endif
#endif
and this
#if
abc
#endif
The corresponding outputs:
[#if s
][s
][#if a
][s
][s
][#endif
][#endif
]
and this
[#if
][abc
][#endif
]
Problem:
All the output lines including #if #endif are surrounded by [] meaning they are matched ONLY by oneNormalInputLine! But I am not expecting this. The preprocessorLineSet should be able to match the #if lines. Why'd I get this result?
This line contains ambiguous alternatives:
inputLines : ( preprocessorLineSet | oneNormalInputLine );
since both can match the #if and #endif. But I am expecting the first alternative should be used rather than the later one. Also note that backtracking is on.
EDIT
The reason my oneNormalInputLine rule accepts everything is that it is difficult to express something not having a specific pattern as #if pattern can be rather complicated:
/***
comments
*/ # /***
comments
*/ if
is a valid pattern. Writing a rule not having this pattern seems difficult.
Your approach is not really robust - I'd suggest you to keep it simple and use the actual language rule, which says that every line that begins with # is a preprocessor directive, and the one that doesn't begin with # isn't. There would be no ambiguity in the grammar using this rule and it would be much simpler to understand.
Now why doesn't your grammar work? The problem is that your preprocesstoLineSet rule can't match anything.
preprocessorLineSet
: ifPart endifLine;
ifPart: ifLine inputLines* ;
It starts by #if ..., then should match other lines, and as the first matching #endif comes, it should match it and finish. However, it doesn't actually do that. inputLines can match pretty much any line (pretty much - it won't match eg. C++'s operators and other non-identifiers), including all preprocessor directives. That means the ifPart rule will match to the end of input and there would be no endifLine left. Note that backtracking has no effect on this, because once ANTLR matches a rule (in this case ifPart, which will succeed on the whole rest of the input, since * is greedy), it will never backtrack into it. ANTLR's rules for backtracking are hairy...
Note that if you made oneNormalLine not match preprocessor directives (eg. it would be something like (nonHash any*| ) CRLF, it would start to work.
Your any_token_except_crlf is causing the ambiguity. You need to fix that (and remove backtrack=true;) by letting that rule match the following:
space-chars;
a '#' followed by anything other than 'if', 'endif' and line breaks;
any char except '#' and line breaks, followed by a 'if' or 'endif'
an identifier.
A small working example (I named the rules a bit differently...):
grammar PPMacro;
options {
output=AST;
}
tokens {
FILE;
}
file
: line+ EOF -> ^(FILE line+)
;
line
: if_stat
| normal_line
;
if_stat
: HASH IF normal_line line* HASH ENDIF -> ^(IF normal_line line*)
;
normal_line
: non_special* CRLF -> non_special*
;
non_special
: WS
| HASH ~(IF | ENDIF | CRLF)
| ~(HASH | CRLF) (IF | ENDIF)
| ID
;
CRLF : '\r'? '\n' ;
WS : (' ' | '\t' | '\f')+;
HASH : '#' ;
IF : 'if' ;
ENDIF : 'endif' ;
ID : ( 'a'..'z'|'A'..'Z'|'0'..'9'| '_')+ ;
This can be tested with the class:
import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;
public class Main {
public static void main(String[] args) throws Exception {
PPMacroLexer lexer = new PPMacroLexer(new ANTLRFileStream("test.cpp"));
PPMacroParser parser = new PPMacroParser(new CommonTokenStream(lexer));
CommonTree tree = (CommonTree)parser.file().getTree();
DOTTreeGenerator gen = new DOTTreeGenerator();
StringTemplate st = gen.toDOT(tree);
System.out.println(st);
}
}
and the test.cpp file that might look like this:
a b
#if s
t
#if a
u
v
#endif
#endif
c
d
which will produce the following AST:
EDIT
I just now saw that you want to account for multi-line comments and spaces between # and if (and endif). You could handle such a thing best in the lexer, like this:
grammar PPMacro;
options {
output=AST;
}
tokens {
FILE;
ENDIF;
}
file
: line+ EOF -> ^(FILE line+)
;
line
: if_stat
| normal_line
;
if_stat
: IF normal_line line* ENDIF -> ^(IF normal_line line*)
;
normal_line
: non_special* CRLF -> non_special*
;
non_special
: WS
| ID
;
IF : '#' NOISE* ('if' | 'endif' {$type=ENDIF;});
CRLF : '\r'? '\n';
WS : (' ' | '\t' | '\f')+;
ID : ('a'..'z' | 'A'..'Z' | '0'..'9' | '_')+;
COMMENT : '/*' .* '*/' {skip();};
fragment NOISE
: '/*' .* '*/'
| WS
;
fragment ENDIF : ;
which will parse the following input:
a b
# /*
comment
*/ if s
t
# if a
u
v
# /*
another
comment */ endif
#endif
c
d
in pretty much the same AST as I posted above.

Resources