My grammar is producing an unexpected result. I am not sure if it is just my bug or some issues with ANTLR'S ambiguous alternatives processing logic.
Here is my grammar :
grammar PPMacro;
options {
language=Java;
backtrack=true;
}
file: (inputLines)+ EOF;
inputLines
: ( preprocessorLineSet | oneNormalInputLine ) ;
oneNormalInputLine #after{System.out.print("["+$text+"]");}
: (any_token_except_crlf)* CRLF ;
preprocessorLineSet
: ifPart endifLine;
ifPart: ifLine inputLines* ;
ifLine #after{System.out.print("{"+$text+"}" );}
: '#' IF (any_token_except_crlf)* CRLF ;
endifLine #after{System.out.print("{"+$text+"}" );}
: '#' ENDIF (any_token_except_crlf)* CRLF ;
any_token_except_crlf: (ANY_ID | WS | '#'|IF|ENDIF);
// just matches everything
CRLF: '\r'? '\n' ;
WS: (' '|'\t'|'\f' )+;
Hash: '#' ;
IF : 'if' ;
ENDIF : 'endif' ;
ANY_ID: ( 'a'..'z'|'A'..'Z'|'0'..'9'| '_')+ ;
Explanation:
It is for parsing a C++ #if ... #endif block
I am trying to recognize nested #if #endif block. This is done by my preprocessorLineSet. It contains a recursive definition to support nested block. oneNormalInputLine is to handle anything not of the #if form. This rule is a match anything rule and actually matches a #if line. But I deliberately put it after the preprocessorLineSet in inputLines. I'm expecting this ordering can prevent it from matching a #if or #endif line. The reason to use a catch-all rule is that I want a rule to accept any other c++ syntax and simply echo them back to the output.
I my test, I just print out everything. Lines matched by preprocessorLineSet should be surrounded by {}, while those matched by oneNormalInputLine should be surrounded by [].
Sample inputs :
#if s
s
#if a
s
s
#endif
#endif
and this
#if
abc
#endif
The corresponding outputs:
[#if s
][s
][#if a
][s
][s
][#endif
][#endif
]
and this
[#if
][abc
][#endif
]
Problem:
All the output lines including #if #endif are surrounded by [] meaning they are matched ONLY by oneNormalInputLine! But I am not expecting this. The preprocessorLineSet should be able to match the #if lines. Why'd I get this result?
This line contains ambiguous alternatives:
inputLines : ( preprocessorLineSet | oneNormalInputLine );
since both can match the #if and #endif. But I am expecting the first alternative should be used rather than the later one. Also note that backtracking is on.
EDIT
The reason my oneNormalInputLine rule accepts everything is that it is difficult to express something not having a specific pattern as #if pattern can be rather complicated:
/***
comments
*/ # /***
comments
*/ if
is a valid pattern. Writing a rule not having this pattern seems difficult.
Your approach is not really robust - I'd suggest you to keep it simple and use the actual language rule, which says that every line that begins with # is a preprocessor directive, and the one that doesn't begin with # isn't. There would be no ambiguity in the grammar using this rule and it would be much simpler to understand.
Now why doesn't your grammar work? The problem is that your preprocesstoLineSet rule can't match anything.
preprocessorLineSet
: ifPart endifLine;
ifPart: ifLine inputLines* ;
It starts by #if ..., then should match other lines, and as the first matching #endif comes, it should match it and finish. However, it doesn't actually do that. inputLines can match pretty much any line (pretty much - it won't match eg. C++'s operators and other non-identifiers), including all preprocessor directives. That means the ifPart rule will match to the end of input and there would be no endifLine left. Note that backtracking has no effect on this, because once ANTLR matches a rule (in this case ifPart, which will succeed on the whole rest of the input, since * is greedy), it will never backtrack into it. ANTLR's rules for backtracking are hairy...
Note that if you made oneNormalLine not match preprocessor directives (eg. it would be something like (nonHash any*| ) CRLF, it would start to work.
Your any_token_except_crlf is causing the ambiguity. You need to fix that (and remove backtrack=true;) by letting that rule match the following:
space-chars;
a '#' followed by anything other than 'if', 'endif' and line breaks;
any char except '#' and line breaks, followed by a 'if' or 'endif'
an identifier.
A small working example (I named the rules a bit differently...):
grammar PPMacro;
options {
output=AST;
}
tokens {
FILE;
}
file
: line+ EOF -> ^(FILE line+)
;
line
: if_stat
| normal_line
;
if_stat
: HASH IF normal_line line* HASH ENDIF -> ^(IF normal_line line*)
;
normal_line
: non_special* CRLF -> non_special*
;
non_special
: WS
| HASH ~(IF | ENDIF | CRLF)
| ~(HASH | CRLF) (IF | ENDIF)
| ID
;
CRLF : '\r'? '\n' ;
WS : (' ' | '\t' | '\f')+;
HASH : '#' ;
IF : 'if' ;
ENDIF : 'endif' ;
ID : ( 'a'..'z'|'A'..'Z'|'0'..'9'| '_')+ ;
This can be tested with the class:
import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;
import org.antlr.stringtemplate.*;
public class Main {
public static void main(String[] args) throws Exception {
PPMacroLexer lexer = new PPMacroLexer(new ANTLRFileStream("test.cpp"));
PPMacroParser parser = new PPMacroParser(new CommonTokenStream(lexer));
CommonTree tree = (CommonTree)parser.file().getTree();
DOTTreeGenerator gen = new DOTTreeGenerator();
StringTemplate st = gen.toDOT(tree);
System.out.println(st);
}
}
and the test.cpp file that might look like this:
a b
#if s
t
#if a
u
v
#endif
#endif
c
d
which will produce the following AST:
EDIT
I just now saw that you want to account for multi-line comments and spaces between # and if (and endif). You could handle such a thing best in the lexer, like this:
grammar PPMacro;
options {
output=AST;
}
tokens {
FILE;
ENDIF;
}
file
: line+ EOF -> ^(FILE line+)
;
line
: if_stat
| normal_line
;
if_stat
: IF normal_line line* ENDIF -> ^(IF normal_line line*)
;
normal_line
: non_special* CRLF -> non_special*
;
non_special
: WS
| ID
;
IF : '#' NOISE* ('if' | 'endif' {$type=ENDIF;});
CRLF : '\r'? '\n';
WS : (' ' | '\t' | '\f')+;
ID : ('a'..'z' | 'A'..'Z' | '0'..'9' | '_')+;
COMMENT : '/*' .* '*/' {skip();};
fragment NOISE
: '/*' .* '*/'
| WS
;
fragment ENDIF : ;
which will parse the following input:
a b
# /*
comment
*/ if s
t
# if a
u
v
# /*
another
comment */ endif
#endif
c
d
in pretty much the same AST as I posted above.
Related
The grammar below is failing to generate a parser with this error:
error(119): Sable.g4::: The following sets of rules are mutually left-recursive [expression]
1 error(s)
The error disappears when I comment out the embedded actions, but comes back when I make them effective.
Why are the actions bringing about such left-recursion that antlr4 cannot handle it?
grammar Sable;
options {}
#header {
package org.sable.parser;
import org.sable.parser.internal.*;
}
sourceFile : statement* EOF;
statement : expressionStatement (SEMICOLON | NEWLINE);
expressionStatement : expression;
expression:
{System.out.println("w");} expression '+' expression {System.out.println("tf?");}
| IDENTIFIER
;
SEMICOLON : ';';
RPARENTH : '(';
LPARENTH : ')';
NEWLINE:
('\u000D' '\u000A')
| '\u000A'
;
WS : WhiteSpaceNotNewline -> channel(HIDDEN);
IDENTIFIER:
(IdentifierHead IdentifierCharacter*)
| ('`'(IdentifierHead IdentifierCharacter*)'`')
;
fragment DecimalDigit :'0'..'9';
fragment IdentifierHead:
'a'..'z'
| 'A'..'Z'
;
fragment IdentifierCharacter:
DecimalDigit
| IdentifierHead
;
// Non-newline whitespaces are defined apart because they carry meaning in
// certain contexts, e.g. within space-aware operators.
fragment WhiteSpaceNotNewline : [\u0020\u000C\u0009u000B\u000C];
UPDATE: the following workaround kind of solves this specific situation, but not the case when init/after-like actions are needed on a lesser scope - per choice, not per rule.
expression
#init {
enable(Token.HIDDEN_CHANNEL);
}
#after {
disable(Token.HIDDEN_CHANNEL);
}:
expression '+' expression
| IDENTIFIER
;
i was trying to code a parser using yacc and lex that count the number of nested loops (while or for).I started the implementation for just while loops.But for some reason the parser gives me an error at the end of a closing brace.
Here is the code.
%{
#include<stdio.h>
/*parser for counting while loops*/
extern int yyerror(char* error);
int while_count=0;
extern int yylex();
%}
%token NUMBER
%token VAR
%token WHILE
%%
statement_list : statement'\n'
| statement_list statement'\n'
;
statement :
while_stmt '\n''{' statement_list '}'
| VAR '=' NUMBER ';'
;
while_stmt :
WHILE '('condition')' {while_count++;}
;
condition :
VAR cond_op VAR
;
cond_op : '>'
| '<'
| '=''='
| '!''='
;
%%
int main(void){
yyparse();
printf("while count:%d\n",while_count);
}
int yyerror(char *s){
printf("Error:%s\n",s);
return 1;
}
what is wrong with that code.And is there a way in yacc to mention optional arguments? like the "\n" after while?
here is the lexer code
%{
#include"y.tab.h"
/*lexer for scanning nested while loops*/
%}
%%
[\t ] ; /*ignore white spaces*/
"while" {return WHILE;}
[a-zA-Z]+ {return VAR;}
[0-9]+ {return NUMBER;}
'$' {return 0;}
'\n' {return '\n' ;}
. {return yytext[0];}
%%
VAR is a variable name with just ascii characters and WHILE is the keyword while.type is not taken into consideration on variable assignments
The problem you seem to be having is with empty loop bodies, not nested loops. As written, your grammar requires at least one statement in the while loop body. You can fix this by allowing empty statement lists:
statement_list: /* empty */
| statement_list statement '\n'
;
You also ask about making newlines optional. The easiest way is to make the lexer simply discard newlines (as whitespace) rather than returning them. Then just get rid of the newlines in the grammar, and newlines can appear between any two tokens and will be ignored.
If you really must have newlines in the grammar for some reason, you can add a rule like:
opt_newlines: /* empty */ | opt_newlines '\n' ;
and then use this rule wherever you want to allow for newlines (replace all the literal '\n' in your grammar.) You have to be careful not to use it redundantly, however. If you do something like:
statement_list: /* empty */
| statement_list statement opt_newlines
;
while_stmt opt_newlines '{' opt_newlines statement_list opt_newlines '}'
you'll get shift/reduce conflicts as newlines before the } in a loop could be either part of the opt_newlines in the while or the opt_newlines in the statement_list. Its pretty easy to deal such conflicts by just removing the redundant opt_newlines.
Background
I have been using ANTLRWorks (V 1.4.3) for a few days now and trying to write a simple Boolean parser. The combined lexer/parser grammar below works well for most of the requirements including support for quoted white-spaced text as operands for a Boolean expression.
Problem
I would like the grammar to work for white-spaced operands without the need of quotes.
Example
For example, expression-
"left right" AND center
should have the same parse tree even after dropping the quotes-
left right AND center.
I have been learning about backtracking, predicates etc but can't seem to find a solution.
Code
Below is the grammar I have got so far. Any feedback on the foolish mistakes is appreciated :).
Lexer/Parser Grammar
grammar boolean_expr;
options {
TokenLabelType=CommonToken;
output=AST;
ASTLabelType=CommonTree;
}
#modifier{public}
#ctorModifier{public}
#lexer::namespace{Org.CSharp.Parsers}
#parser::namespace{Org.CSharp.Parsers}
public
evaluator
: expr EOF
;
public
expr
: orexpr
;
public
orexpr
: andexpr (OR^ andexpr)*
;
public
andexpr
: notexpr (AND^ notexpr)*
;
public
notexpr
: (NOT^)? atom
;
public
atom
: word | LPAREN! expr RPAREN!
;
public
word
: QUOTED_TEXT | TEXT
;
/*
* Lexer Rules
*/
LPAREN
: '('
;
RPAREN
: ')'
;
AND
: 'AND'
;
OR
: 'OR'
;
NOT
: 'NOT'
;
WS
: ( ' ' | '\t' | '\r' | '\n') {$channel=HIDDEN;}
;
QUOTED_TEXT
: '"' (LETTER | DIGIT | ' ' | ',' | '-')+ '"'
;
TEXT
: (LETTER | DIGIT)+
;
/*
Fragment lexer rules can be used by other lexer rules, but do not return tokens by themselves
*/
fragment DIGIT
: ('0'..'9')
;
fragment LOWER
: ('a'..'z')
;
fragment UPPER
: ('A'..'Z')
;
fragment LETTER
: LOWER | UPPER
;
Simply let TEXT in your atom rule match once or more: TEXT+. When it matches a TEXT token more than once, you'll also want to create a custom root node for these TEXT tokens (I added an imaginary token called WORD in the grammar below).
grammar boolean_expr;
options {
output=AST;
}
tokens {
WORD;
}
evaluator
: expr EOF
;
...
word
: QUOTED_TEXT
| TEXT+ -> ^(WORD TEXT+)
;
...
Your input "left right AND center" would now be parsed as follows:
I am using ANTLR to create an and/or parser+evaluator. Expressions will have the format like:
x eq 1 && y eq 10
(x lt 10 && x gt 1) OR x eq -1
I was reading this post on logic expressions in ANTLR Looking for advice on project. Parsing logical expression and I found the grammar posted there a good start:
grammar Logic;
parse
: expression EOF
;
expression
: implication
;
implication
: or ('->' or)*
;
or
: and ('&&' and)*
;
and
: not ('||' not)*
;
not
: '~' atom
| atom
;
atom
: ID
| '(' expression ')'
;
ID : ('a'..'z' | 'A'..'Z')+;
Space : (' ' | '\t' | '\r' | '\n')+ {$channel=HIDDEN;};
However, while getting a tree from the parser works for expressions where the variables are just one character (ie, "(A || B) AND C", I am having a hard time adapting this to my case (in the example "x eq 1 && y eq 10" I'd expect one "AND" parent and two children, "x eq 1" and "y eq 10", see the test case below).
#Test
public void simpleAndEvaluation() throws RecognitionException{
String src = "1 eq 1 && B";
LogicLexer lexer = new LogicLexer(new ANTLRStringStream(src));
LogicParser parser = new LogicParser(new CommonTokenStream(lexer));
CommonTree tree = (CommonTree)parser.parse().getTree();
assertEquals("&&",tree.getText());
assertEquals("1 eq 1",tree.getChild(0).getText());
assertEquals("a neq a",tree.getChild(1).getText());
}
I believe this is related with the "ID". What would the correct syntax be?
For those interested, I made some improvements in my grammar file (see bellow)
Current limitations:
only works with &&/||, not AND/OR (not very problematic)
you can't have spaces between the parenthesis and the &&/|| (I solve that by replacing " (" with ")" and ") " with ")" in the source String before feeding the lexer)
grammar Logic;
options {
output = AST;
}
tokens {
AND = '&&';
OR = '||';
NOT = '~';
}
// parser/production rules start with a lower case letter
parse
: expression EOF! // omit the EOF token
;
expression
: or
;
or
: and (OR^ and)* // make `||` the root
;
and
: not (AND^ not)* // make `&&` the root
;
not
: NOT^ atom // make `~` the root
| atom
;
atom
: ID
| '('! expression ')'! // omit both `(` and `)`
;
// lexer/terminal rules start with an upper case letter
ID
:
(
'a'..'z'
| 'A'..'Z'
| '0'..'9' | ' '
| SYMBOL
)+
;
SYMBOL
:
('+'|'-'|'*'|'/'|'_')
;
ID : ('a'..'z' | 'A'..'Z')+;
states that an identifier is a sequence of one or more letters, but does not allow any digits. Try
ID : ('a'..'z' | 'A'..'Z' | '0'..'9')+;
which will allow e.g. abc, 123, 12ab, and ab12. If you don't want the latter types, you'll have to restructure the rule a little bit (left as a challenge...)
In order to accept arbitrarily many identifiers, you could define atom as ID+ instead of ID.
Also, you will likely need to specify AND, OR, -> and ~ as tokens so that, as #Bart Kiers says, the first two won't get classified as ID, and so that the latter two will get recognized at all.
I am quite new to ANTLR, so this is likely a simple question.
I have defined a simple grammar which is supposed to include arithmetic expressions with numbers and identifiers (strings that start with a letter and continue with one or more letters or numbers.)
The grammar looks as follows:
grammar while;
#lexer::header {
package ConFreeG;
}
#header {
package ConFreeG;
import ConFreeG.IR.*;
}
#parser::members {
}
arith:
term
| '(' arith ( '-' | '+' | '*' ) arith ')'
;
term returns [AExpr a]:
NUM
{
int n = Integer.parseInt($NUM.text);
a = new Num(n);
}
| IDENT
{
a = new Var($IDENT.text);
}
;
fragment LOWER : ('a'..'z');
fragment UPPER : ('A'..'Z');
fragment NONNULL : ('1'..'9');
fragment NUMBER : ('0' | NONNULL);
IDENT : ( LOWER | UPPER ) ( LOWER | UPPER | NUMBER )*;
NUM : '0' | NONNULL NUMBER*;
fragment NEWLINE:'\r'? '\n';
WHITESPACE : ( ' ' | '\t' | NEWLINE )+ { $channel=HIDDEN; };
I am using ANTLR v3 with the ANTLR IDE Eclipse plugin. When I parse the expression (8 + a45) using the interpreter, only part of the parse tree is generated:
Why does the second term (a45) not get parsed? The same happens if both terms are numbers.
You'll want to create a parser rule that has an EOF (end of file) token in it so that the parser will be forced to go through the entire token stream.
Add this rule to your grammar:
parse
: arith EOF
;
and let the interpreter start at that rule instead of the arith rule: