My graduate student and I are working on a training compiler, which we will use to teach students at the subject "Compilers and Interpreters".
The input program language is a limited subset of the Java language and the compiler implementation language is Java.
The grammar of the input language syntax is LL(1), because it is easier to be understood and implemented by students. We have the following general problem in the parser implementation. How to differentiate identifier from function call during the parsing?
For example we may have:
b = sum(10,5) //sum is a function call
or
b = a //a is an identifier
In both cases after the = symbol we have an identifier.
Is it possible to differentiate what kind of construct (a function call or an identifier) we have after the equality symbol =?
May be it is not possible in LL(1) parser, as we can look only 1 symbol ahead? If this is true, how do you recommend to define the function call in the grammar? Maybe some additional symbol in front of the function call is necessary, e.g. b = #sum(10,5)?
Do You think this symbol would be confusing for students? What kind of symbol for the function call would be proper?
You indeed can't have separate rules for function calls and variables in an LL(1) grammar because that would require additional lookahead. The common solution to this is to combine them into one rule that matches an identifier, optionally followed by an argument list:
primary_expression ::= ID ( "(" expression_list ")" )?
| ...
In a language where a function can be an arbitrary expression, not just an identifier, you'll want to treat it just like any other postfix operator:
postfix_expression ::= primary_expression postfix_operator*
postfix_operator ::= "++"
| "--"
| "[" expression "]"
| "(" expression_list ")"
Related
typename ::= typename DOT ID.
typename ::= ID.
lvalue ::= lvalue DOT lvalue2.
lvalue ::= lvalue2.
lvalue2 ::= ID LSQB expr RSQB. // LSQB & RSQB: left & right square bracket. i.e. [ ].
lvalue2 ::= ID.
typename is a rule for the names of types. It matches the following code:
ClassA
package_a.ClassA
while lvalue is a rule for left values. It matches the following code:
varA
varB.C
varD.E[i].F
Now the 2 rules conflicts with each other. Maybe it is because lvalue can also match package_a.ClassA?
How can I solve this?
You can't resolve this issue grammatically because your syntax is ambiguous. a.b = 3 is valid if a.b is a member of a, and invalid if a.b is a type, but the semantics of a.b cannot be determined by the syntax.
You could resolve this in a fairly messy way if you have some way to figure that out in the lexer (which will probably involve some kind of lexical feedback, since the lexer would presumably need access to the symbol table in order to provide that information). Then the lexer could use two different token types for IDs, based on whether or not they are type names.
But probably the best option is to either abandon the idea of distinguishing grammatically between lvalues and rvalues, and or to assume that all selection operations (a.b) produce lvalues, and then to validate the use of an expression as an lvalue in the semantic action or some subsequent semantic analysis.
I have the following grammar rules defined to cover tuples of the form: (a), (a,), (a,b), (a,b,) and so on. However, antlr3 gives the warning:
"Decision can match input such as "COMMA" using multiple alternatives: 1, 2
I believe this means that my grammar is not LL(1). This caught me by surprise as, based on my extremely limited understanding of this topic, the parser would only need to look one token ahead from (COMMA)? to ')' in order to know which comma it was on.
Also based on the discussion I found here I am further confused: Amend JSON - based grammar to allow for trailing comma
And their source code here: https://github.com/doctrine/annotations/blob/1.13.x/lib/Doctrine/Common/Annotations/DocParser.php#L1307
Is this because of the kind of parser that antlr is trying to generate and not because my grammar isn't LL(1)? Any insight would be appreciated.
options {k=1; backtrack=no;}
tuple : '(' IDENT (COMMA IDENT)* (COMMA)? ')';
DIGIT : '0'..'9' ;
LOWER : 'a'..'z' ;
UPPER : 'A'..'Z' ;
IDENT : (LOWER | UPPER | '_') (LOWER | UPPER | '_' | DIGIT)* ;
edit: changed typo in tuple: ... from (IDENT)? to (COMMA)?
Note:
The question has been edited since this answer was written. In the original, the grammar had the line:
tuple : '(' IDENT (COMMA IDENT)* (IDENT)? ')';
and that's what this answer is referring to.
That grammar works without warnings, but it doesn't describe the language you intend to parse. It accepts, for example, (a, b c) but fails to accept (a, b,).
My best guess is that you actually used something like the grammars in the links you provide, in which the final optional element is a comma, not an identifier:
tuple : '(' IDENT (COMMA IDENT)* (COMMA)? ')';
That does give the warning you indicate, and it won't match (a,) (for example), because, as the warning says, the second alternative has been disabled.
LL(1) as a property of formal grammars only applies to grammars with fixed right-hand sides, as opposed to the "Extended" BNF used by many top-down parser generators, including Antlr, in which a right-hand side can be a set of possibilities. It's possible to expand EBNF using additional non-terminals for each subrule (although there is not necessarily a canonical expansion, and expansions might differ in their parsing category). But, informally, we could extend the concept of LL(k) by saying that in every EBNF right-hand side, at every point where there is more than one alternative, the parser must be able to predict the appropriate alternative looking only at the next k tokens.
You're right that the grammar you provide is LL(1) in that sense. When the parser has just seen IDENT, it has three clear alternatives, each marked by a different lookahead token:
COMMA ↠ predict another repetition of (COMMA IDENT).
IDENT ↠ predict (IDENT).
')' ↠ predict an empty (IDENT)?.
But in the correct grammar (with my modification above), IDENT is a syntax error and COMMA could be either another repetition of ( COMMA IDENT ), or it could be the COMMA in ( COMMA )?.
You could change k=1 to k=2, thereby allowing the parser to examine the next two tokens, and if you did so it would compile with no warnings. In effect, that grammar is LL(2).
You could make an LL(1) grammar by left-factoring the expansion of the EBNF, but it's not going to be as pretty (or as easy for a reader to understand). So if you have a parser generator which can cope with the grammar as written, you might as well not worry about it.
But, for what it's worth, here's a possible solution:
tuple : '(' idents ')' ;
idents : IDENT ( COMMA ( idents )? )? ;
Untested because I don't have a working Antlr3 installation, but it at least compiles the grammar without warnings. Sorry if there is a problem.
It would probably be better to use tuple : '(' (idents)? ')'; in order to allow empty tuples. Also, there's no obvious reason to insist on COMMA instead of just using ',', assuming that '(' and ')' work as expected on Antlr3.
What is the correct grammar for a standard member expression?
E.g. the ast from the code:
test.test.function()
would be
MemberExpression("test", MemberExpression("test", MethodCall("function")))
And likewise for a variable:
test.test.test.variable
MemberExpression("test", MemberExpression("test", MemberExpression("test", Variable("variable"))))
Depends on the language, surely :-) But it's pretty straight-up in most grammars (see below).
One comment, though. As indicated by the grammars below, member access (like function calls and, usually, subscripting) acts like a postfix operator; the symbol after the dot (or arrow, in C-like languages) is a symbol representing a member name. It is not an expression; the only expression in the member lookup is on the left-hand side of the operator. So a.b.c should correspond to an AST node something like:
MemberLookup(MemberLookup(Variable("a"), "b"), "c")
and a.b.func(2, c) should be turned into:
MethodCall(MemberLookup(Variable("a"), "b"),
"func",
List(Number(2), Variable("c")))
or, perhaps,
Apply(MemberLookup(MemberLookup(Variable("a"), "b"), "func"),
List(Number(2), Variable("c"))
(The difference has to do with the implicit self/this argument; there are various strategies for handling this. Contrast Java, Python and Lua for three completely different strategies.)
Anyway, here's a couple of simple grammar fragments:
C
Here's an excerpt from the C grammar (as found in Appendix A of the C standard:
postfix-expression:
primary-expression
postfix-expression '[' expression ']'
postfix-expression '(' argument-expression-listopt ')'
postfix-expression '.' identifier
postfix-expression '->' identifier
postfix-expression '++'
postfix-expression '--'
I included more than just the member access functions, because it shows that .identifier and ->identifier are handled just like any other postfix operator, which is a useful insight. The same production also includes two postfix bracketed operators, subscripting ([...]) and function call ((...)), which seem relevant here. But I left out compound literals (which I would have put into primary-expression).
Python
The comparable excerpt from the Python 3.9 docs:
primary:
primary '.' NAME
primary '(' [arguments] ')'
primary '[' slices ']'
primary genexp
atom
I'm currently writing a simple grammar that requires operator precedence and mixed associativities in one expression. An example expression would be a -> b ?> C ?> D -> e, which should be parsed as (a -> (((b ?> C) ?> D) -> e). That is, the ?> operator is a high-precedence left-associative operator wheras the -> operator is a lower-precedence right-associative operator.
I'm prototyping the grammar in ANTLR 3.5.1 (via ANTLRWorks 1.5.2) and find that it can't handle the following grammar:
prog : expr EOF;
expr : term '->' expr
| term;
term : ID rest;
rest : '?>' ID rest
| ;
It produces rule expr has non-LL(*) decision due to recursive rule invocations reachable from alts 1,2 error.
The term and rest productions work fine in isolation when I tested it , so I assumed this happened because the parser is getting confused by expr. To get around that, I did the following refactor:
prog : expr EOF;
expr : term exprRest;
exprRest
: '->' expr
| ;
term : ID rest;
rest : DU ID rest
| ;
This works fine. However, because of this refactor I now need to check for empty exprRest nodes in the output parse tree, which is non-ideal. Is there a way to make ANTLR work around the ambiguity in the initial declaration of expr? I would of assumed that the generated parser would fully match term and then do a lookahead search for "->" and either continue parsing or return the lone term. What am I missing?
As stated, the problem is in this rule:
expr : term '->' expr
| term;
The problematic part is the term which is common to both alternatives.
LL(1) grammar doesn't allow this at all (unless term only matches zero tokens - but such rules would be pointless), because it cannot decide which alternative to use with only being able to see one token ahead (that's the 1 in LL(1)).
LL(k) grammar would only allow this if the term rule could match at most k - 1 tokens.
LL(*) grammar which ANTLR 3.5 uses does some tricks that allows it to handle rules that match any number of tokens (ANTLR author calls this "variable look-ahead").
However, one thing that these tricks cannot handle is if the rule is recursive, i.e. if it or any rules it invokes reference itself in any way (direct or indirect) - and that is exactly what your term rule does:
term : ID rest;
rest : '?>' ID rest
| ;
- the rule rest, referenced from term, recursively references itself. Thus, the error message
rule expr has non-LL(*) decision due to recursive rule invocations ...
The way to solve this limitation of LL grammars is called left-factoring:
expr : term
( '->' expr )?
;
What I did here is said "match term first" (since you want to match it in both alternatives, there's no point in deciding which one to match it in), then decide whether to match '->' expr (this can be decided just by looking at the very next token - if it's ->, use it - so this is even LL(1) decision).
This is very similar to what you came to as well, but the parse tree should look very much like you intended with the original grammar.
I have written a lexer and parser to analyze linear algebra statements. Each statement consists of one or more expressions followed by one or more declarations. I am using menhir and OCaml to write the lexer and parser.
For example:
Ax = b, where A is invertible.
This should be read as A * x = b, (A, invertible)
In an expression all ids must be either an uppercase or lowercase symbol. I would like to overload the multiplication operator so that the user does not have to type in the '*' symbol.
However, since the lexer also needs to be able to read strings (such as "invertible" in this case), the "Ax" portion of the expression is sent over to the parser as a string. This causes a parser error since no strings should be encountered in the expression portion of the statement.
Here is the basic idea of the grammar
stmt :=
| expr "."
| decl "."
| expr "," decl "."
expr :=
| term
| unop expr
| expr binop expr
term :=
| <int> num
| <char> id
| "(" expr ")"
decl :=
| id "is" kinds
kinds :=
| <string> kind
| kind "and" kinds
Is there some way to separate the individual characters and tell the parser that they should be treated as multiplication? Is there a way to change the lexer so that it is smart enough to know that all character clusters before a comma are ids and all clusters after should be treated as strings?
It seems to me you have two problems:
You want your lexer to treat sequences of characters differently in different places.
You want multiplication to be indicated by adjacent expressions (no operator in between).
The first problem I would tackle in the lexer.
One question is why you say you need to use strings. This implies that there is a completely open-ended set of things you can say. It might be true, but if you can limit yourself to a smallish number, you can use keywords rather than strings. E.g., invertible would be a keyword.
If you really want to allow any string at all in such places, it's definitely still possible to hack a lexer so that it maintains a state describing what it has seen, and looks ahead to see what's coming. If you're not required to adhere to a pre-defined grammar, you could adjust your grammar to make this easier. (E.g., you could use commas for only one purpose.)
For the second problem, I'd say you need to add adjacency to your grammar. I.e., your grammar needs a rule that says something like term := term term. I suspect it's tricky to get this to work correctly, but it does work in OCaml (where adjacent expressions represent function application) and in awk (where adjacent expressions represent string concatenation).