Grammar: http://pastebin.com/ef2jt8Rg
y.output: http://pastebin.com/AEKXrrRG
I don't know where is those conflicts, someone can help me with this?
The y.output file tells you exactly where the conflicts are. The first one is in state 4, so if you go down to look at state 4, you see:
state 4
99 compound_statement: '{' . '}'
100 | '{' . statement_list '}'
IDENTIFIER shift, and go to state 6
:
IDENTIFIER [reduce using rule 1 (threat_as_ref)]
IDENTIFIER [reduce using rule 2 (func_call_start)]
This is telling you that in this state (parsing a compound_statement, having seen a {), and looking at the next token being IDENTIFIER, there are 3 possible things it could do -- shift the token (which would be the beginning of a statement_list), reduce the threat_as_ref empty production, or reduce the func_call_start empty production.
The brackets tell you that it has decided to never do those actions -- the default "prefer shift over reduce" conflict resolution means that it will always do the shift.
The problem with your grammar is these empty rules threat_as_ref and func_call_start -- they need to be reduced BEFORE shifting the IDENTIFIER, but in order to know if they're valid, the parser would need to see the tokens AFTER the identifer. func_call_start should only be reduced if this is the beginning of the function call (which depends on there being a ( after the IDENTIFIER.) So the parser needs more lookahead to deal with your gramar. In your specific case, you grammar is LALR(2) (2 token lookahead would suffice), but not LALR(1), so bison can't deal with it.
Now you could fix it by just getting rid of those empty rules -- func_call_start has no action at all, and the action for threat_as_ref could be moved into the action for variable, but if you want those rules in the future that may be a problem.
(1) I see at least one thing that looks odd. Your productions for expression_statement are similar to those for postfix_statement, but not quite the same. They don't have the '(' and ')' tokens:
expression_statement
: ';'
| expression ';'
| func_call_start IDENTIFIER { ras_parse_variable_psh($2); aFree($2); } func_call_end ';'
| func_call_start IDENTIFIER { ras_parse_variable_psh($2); aFree($2); } argument_expression_list func_call_end ';'
;
Since an expression can be a primary_expression, which can be an IDENTIFIER, and since func_call_start and func_call_end are epsilon (null) productions, when presented with the input
foo;
the parser has to decide whether to apply
expression_statement : expression ';'
or
expression_statement : func_call_start IDENTIFIER { ras_parse_variable_psh($2); aFree($2); } func_call_end ';'
(2) Also, I'm not certain of this, but I suspect the epsilon non-terminal threat_as_ref might be causing you some trouble. I have not traced it through, but there may be a case where the parser has to decide whether something is a variable_ref or a variable.
Related
I have some grammar here in Bison: https://pastebin.com/raw/dA2bypFR.
It's fairly long but not very complex.
The problem is that after a call, it won't accept anything other than ; e.g a(b)(c) and is invalid, a(b).c is invalid, which both only accept a semicolon after the closing parenthesis.
a(b)+c is fine though.
I tried separating call_or_getattr into 2 where . has higer precedence than ( but this meant that a().b was invalid grammar.
I also tried putting call and getattr into the definition for basic_operand but this resulted in a 536 shift/reduce errors.
Your last production reads as follows (without the actions, which are an irrelevant distraction):
call_or_getattr:
basic_operand
| basic_operand '(' csv ')'
| basic_operand '.' T_ID
So those are postfix operators whose argument must be a basic_operand. In a(b)(c), the (c) argument list is not being applied to a basic_operand, so the grammar isn't going to match it.
What you were looking for, I suppose, is:
call_or_getattr:
basic_operand
| call_or_getattr '(' csv ')'
| call_or_getattr '.' T_ID
This is, by the way, very similar to the way you write productions for a binary operator. (Of course, the binary operator has a right-hand operand.)
I am writing a parser using Bison, but can't seem to get the grammar correct.
There are two conflicts:
Here are some of the rules used around conflict one:
program : function END_OF_FILE {return 0;}
formal_parameters : OPEN_PAREN formal_parameter list_E_fparameter CLOSE_PAREN | OPEN_PAREN CLOSE_PAREN
formal_parameter : expression_parameter | function_parameter
function : return_options IDENTIFIER formal_parameters block
function_parameter : return_options IDENTIFIER formal_parameters
expression_parameter : VAR identifier_list IDENTIFIER | identifier_list IDENTIFIER
variable_creation : identifier_list COLON type SEMI_COLON
labels : LABELS identifier_list SEMI_COLON
list_E_identifiers : list_E_identifiers COMMA IDENTIFIER |
identifier_list : IDENTIFIER list_E_identifiers
return_options : VOID | IDENTIFIER
State 12 conflicts: 1 reduce/reduce
state 12
56 identifier_list: IDENTIFIER . list_E_identifiers
60 return_options: IDENTIFIER .
102 list_E_identifiers: . list_E_identifiers COMMA IDENTIFIER
103 | .
COMMA reduce using rule 103 (list_E_identifiers)
IDENTIFIER reduce using rule 60 (return_options)
IDENTIFIER [reduce using rule 103 (list_E_identifiers)]
$default reduce using rule 60 (return_options)
list_E_identifiers go to state 23
State 64 conflicts: 1 shift/reduce
state 64
8 body: OPEN_BRACE list_E_statement . CLOSE_BRACE
17 statement: . opt_declaration unlabeled_statement
18 | . compound
31 compound: . OPEN_BRACE list_NE_unlstatement CLOSE_BRACE
73 opt_declaration: . IDENTIFIER COLON
74 | .
94 list_E_statement: list_E_statement . statement
CLOSE_BRACE shift, and go to state 68
IDENTIFIER shift, and go to state 69
OPEN_BRACE shift, and go to state 70
IDENTIFIER [reduce using rule 74 (opt_declaration)]
$default reduce using rule 74 (opt_declaration)
statement go to state 71
compound go to state 72
opt_declaration go to state 73
Can anyone help me? I've looked at http://www.gnu.org/software/bison/manual/html_node/Understanding.html but can't understand what this means.
I can post the full grammar if that would help.
Thank you!
The second conflict is a classic problem with "optional" elements. It's very tempting to write a rule for optional labels as you have done it, but the fact that optional_label could produce nothing forces the parser to try to make a decision before it has enough information.
LR parsers must "reduce" (recognize) a non-terminal before absorbing any further tokens. They can lookahead at the next k tokens (the next 1 token for an LR(1) parser, which is what bison generates), but they can't tentatively use the token and later go back and do the reduction.
So when the parser is at the point where the next token, which is an identifier, should start a statement, it might be looking at a statement which starts with an identifier, or it might be looking at a label which starts with an identifier. It could tell the difference by looking at the colon which follows the identifier (if any) but it can't see that far ahead.
Now, if it weren't for the fact that it is required to reduce either an empty optional_declaration or one containing a label, there would not be a problem. If you had written something like this:
statement: basic_statement | compound
basic_statement: unlabeled_statement | declaration unlabeled_statement
declaration: IDENTIFIER COLON
then the parser would not have to make a decision when it sees the identifier. It only has to make a decision when it reaches the end of a production; it is perfectly capable of pressing forward when there are two possible productions to complete. But when you force it to recognize an optional label, then it has to know whether the label was not there in order to reduce (recognize) the empty production.
For the first conflict, we can see from the output that there is some context in which the lookahead symbol is IDENTIFIER and you could have either a return_options or an identifier_list. Since both of those productions can produce a single IDENTIFIER, the parser will not know which one to reduce.
With the actual grammar available, it is easy to find the context in which both return_options IDENTIFIER and identifier_list IDENTIFIER are possible:
formal_parameter : expression_parameter | function_parameter
expression_parameter: identifier_list IDENTIFIER
function_parameter : return_options IDENTIFIER …
That grammar is not ambiguous. If IDENTIFIER IDENTIFIER is the start of function_parameter, then it must be followed by a (; if it is an expression_parameter, then it must be followed by either , or ). But that's the second next token, which means you'd need an LR(2) parser.
So I will give my usual advice on handling LR(2) grammars. It is possible to rewrite an LR(k) grammar as an LR(1) grammar, regardless of the value of k, but the result is usually bloated and ugly. So if you are using bison and you are willing to live with the possibility that action evaluations could be slightly delayed, then your easiest solution is to ask bison to generate a GLR parser. Often, just adding %glr-parser to the options section is enough.
It's worth noting that your grammar seems to be an uneasy mix between C and Pascal-like syntaxes. In C, the first token in a parameter is always a type; either the return type of a function, or the type of the following identifier. In Pascal, the last token in a parameter is the type. But in your grammar, sometimes the first token is the type and sometimes it's the last token. In a certain sense, it is this inconsistency which leads to the awkwardness in the grammar.
(Pascal has a lot more punctuation: the type is always preceded by a colon and a function parameter is preceded by the word function. These extra tokens are not needed to make the grammar work, but it can be argued that they make the syntax easier to read by humans.)
I'm writing a grammar for a toy language in Yacc (the one packaged with Go) and I have an expected shift-reduce conflict due to the following pseudo-issue. I have to distilled the problem grammar down to the following.
start:
stmt_list
expr:
INT | IDENT | lambda | '(' expr ')' { $$ = $2 }
lambda:
'(' params ')' '{' stmt_list '}'
params:
expr | params ',' expr
stmt:
/* empty */ | expr
stmt_list:
stmt | stmt_list ';' stmt
A lambda function looks something like this:
map((v) { v * 2 }, collection)
My parser emits:
conflicts: 1 shift/reduce
Given the input:
(a)
It correctly parses an expr by the '(' expr ')' rule. However given an input of:
(a) { a }
(Which would be a lambda for the identity function, returning its input). I get:
syntax error: unexpected '{'
This is because when (a) is read, the parser is choosing to reduce it as '(' expr ')', rather than consider it to be '(' params ')'. Given this conflict is a shift-reduce and not a reduce-reduce, I'm assuming this is solvable. I just don't know how to structure the grammar to support this syntax.
EDIT | It's ugly, but I'm considering defining a token so that the lexer can recognize the ')' '{' sequence and send it through as a single token to resolve this.
EDIT 2 | Actually, better still, I'll make lambdas require syntax like ->(a, b) { a * b} in the grammar, but have the lexer emit the -> rather than it being in the actual source code.
Your analysis is indeed correct; although the grammar is not ambiguous, it is impossible for the parser to decide with the input reduced to ( <expr> and with lookahead ) whether or not the expr should be reduced to params before shifting the ) or whether the ) should be shifted as part of a lambda. If the next token were visible, the decision could be made, so the grammar LR(2), which is outside of the competence of go/yacc.
If you were using bison, you could easily solve this problem by requesting a GLR parser, but I don't believe that go/yacc provides that feature.
There is an LR(1) grammar for the language (there is always an LR(1) grammar corresponding to any LR(k) grammar for any value of k) but it is rather annoying to write by hand. The essential idea of the LR(k) to LR(1) transformation is to shift the reduction decisions k-1 tokens forward by accumulating k-1 tokens of context into each production. So in the case that k is 2, each production P: N → α will be replaced with productions TNU → Tα U for each T in FIRST(α) and each U in FOLLOW(N). [See Note 1] That leads to a considerable blow-up of non-terminals in any non-trivial grammar.
Rather than pursuing that idea, let me propose two much simpler solutions, both of which you seem to be quite close to.
First, in the grammar you present, the issue really is simply the need for a two-token lookahead when the two tokens are ){. That could easily be detected in the lexer, and leads to a solution which is still hacky but a simpler hack: Return ){ as a single token. You need to deal with intervening whitespace, etc., but it doesn't require retaining any context in the lexer. This has the added bonus that you don't need to define params as a list of exprs; they can just be a list of IDENT (if that's relevant; a comment suggests that it isn't).
The alternative, which I think is a bit cleaner, is to extend the solution you already seem to be proposing: accept a little too much and reject the errors in a semantic action. In this case, you might do something like:
start:
stmt_list
expr:
INT
| IDENT
| lambda
| '(' expr_list ')'
{ // If $2 has more than one expr, report error
$$ = $2
}
lambda:
'(' expr_list ')' '{' stmt_list '}'
{ // If anything in expr_list is not a valid param, report error
$$ = make_lambda($2, $4)
}
expr_list:
expr | expr_list ',' expr
stmt:
/* empty */ | expr
stmt_list:
stmt | stmt_list ';' stmt
Notes
That's only an outline; the complete algorithm includes the mechanism to recover the original parse tree. If k is greater than 2 then T and U are strings the the FIRSTk-1 and FOLLOWk-1 sets.
If it really is a shift-reduce conflict, and you want only the shift behavior, your parser generator may give you a way to prefer a shift vs. a reduce. This is classically how the conflict for grammar rules for "if-then-stmt" and "if-then-stmt-else-stmt" is resolved, when the if statement can also be a statement.
See http://www.gnu.org/software/bison/manual/html_node/Shift_002fReduce.html
You can get this effect two ways:
a) Count on the accidental behavior of the parsing engine.
If an LALR parser handles shifts first, and then reductions if there are no shifts, then you'll get this "prefer shift" for free. All the parser generator has to do is built the parse tables anyway, even if there is a detected conflict.
b) Enforce the accidental behavior. Design (or a get a) parser generator to accept "prefer shift on token T". Then one can supress the ambiguity. One still have to implement the parsing engine as in a) but that's pretty easy.
I think this is easier/cleaner than abusing the lexer to make strange tokens (and that doesn't always work anyway).
Obviously, you could make a preference for reductions to turn it the other way. With some extra hacking, you could make shift-vs-reduce specific the state in which the conflict occured; you can even make it specific to the pair of conflicting rules but now the parsing engine needs to keep preference data around for nonterminals. That still isn't hard. Finally, you could add a predicate for each nonterminal which is called when a shift-reduce conflict is about to occur, and it have it provide a decision.
The point is you don't have to accept "pure" LALR parsing; you can bend it easily in a variety of ways, if you are willing to modify the parser generator/engine a little bit. This gives a really good reason to understand how these tools work; then you can abuse them to your benefit.
I've been trying to tackle a seemingly simple shift/reduce conflict with no avail. Naturally, the parser works fine if I just ignore the conflict, but I'd feel much safer if I reorganized my rules. Here, I've simplified a relatively complex grammar to the single conflict:
statement_list
: statement_list statement
|
;
statement
: lvalue '=' expression
| function
;
lvalue
: IDENTIFIER
| '(' expression ')'
;
expression
: lvalue
| function
;
function
: IDENTIFIER '(' ')'
;
With the verbose option in yacc, I get this output file describing the state with the mentioned conflict:
state 2
lvalue -> IDENTIFIER . (rule 5)
function -> IDENTIFIER . '(' ')' (rule 9)
'(' shift, and go to state 7
'(' [reduce using rule 5 (lvalue)]
$default reduce using rule 5 (lvalue)
Thank you for any assistance.
The problem is that this requires 2-token lookahead to know when it has reached the end of a statement. If you have input of the form:
ID = ID ( ID ) = ID
after parser shifts the second ID (lookahead is (), it doesn't know whether that's the end of the first statement (the ( is the beginning of a second statement), or this is a function. So it shifts (continuing to parse a function), which is the wrong thing to do with the example input above.
If you extend function to allow an argument inside the parenthesis and expression to allow actual expressions, things become worse, as the lookahead required is unbounded -- the parser needs to get all the way to the second = to determine that this is not a function call.
The basic problem here is that there's no helper punctuation to aid the parser in finding the end of a statement. Since text that is the beginning of a valid statement can also appear in the middle of a valid statement, finding statement boundaries is hard.
I'm building a parser for a language I've designed, in which type names start with an upper case letter and variable names start with a lower case letter, such that the lexer can tell the difference and provide different tokens. Also, the string 'this' is recognised by the lexer (it's an OOP language) and passed as a separate token. Finally, data members can only be accessed on the 'this' object, so I built the grammar as so:
%token TYPENAME
%token VARNAME
%token THIS
%%
start:
Expression
;
Expression:
THIS
| THIS '.' VARNAME
| Expression '.' TYPENAME
;
%%
The first rule of Expression allows the user to pass 'this' around as a value (for example, returning it from a method or passing to a method call). The second is for accessing data on 'this'. The third rule is for calling methods, however I've removed the brackets and parameters since they are irrelevant to the problem. The originally grammar was clearly much larger than this, however this is the smallest part that generates the same error (1 Shift/Reduce conflict) - I isolated it into its own parser file and verified this, so the error has nothing to do with any other symbols.
As far as I can see, the grammar given here is unambiguous and so should not produce any errors. If you remove any of the three rules or change the second rule to
Expression '.' VARNAME
there is no conflict. In any case, I probably need someone to state the obvious of why this conflict occurs and how to resolve it.
The problem is that the grammar can only look one ahead. Therefore when you see a THIS then a ., are you in line 2(Expression: THIS '.' VARNAME) or line 3 (Expression: Expression '.' TYPENAME, via a reduction according to line 1).
The grammar could reduce THIS. to Expression. and then look for a TYPENAME or shift it to THIS. and look for a VARNAME, but it has to decide when it gets to the ..
I try to avoid y.output but sometimes it does help. I looked at the file it produced and saw.
state 1
2 Expression: THIS. [$end, '.']
3 | THIS . '.' VARNAME
'.' shift, and go to state 4
'.' [reduce using rule 2 (Expression)]
$default reduce using rule 2 (Expression)
Basically it is saying it sees '.' and can reduce or it can shift. Reduce makes me anrgu sometimes because they are hard to fine. The shift is rule 3 and is obvious (but the output doesnt mention the rule #). The reduce where it see's '.' in this case is the line
| Expression '.' TYPENAME
When it goes to Expression it looks at the next letter (the '.') and goes in. Now it sees THIS | so when it gets to the end of that statement it expects '.' when it leaves or an error. However it sees THIS '.' while its between this and '.' (hence the dot in the out file) and it CAN reduce a rule so there is a path conflict. I believe you can use %glr-parser to allow it to try both but the more conflicts you have the more likely you'll either get unexpected output or an ambiguity error. I had ambiguity errors in the past. They are annoying to deal with especially if you dont remember what rule caused or affected them. it is recommended to avoid conflicts.
I highly recommend this book before attempting to use bison.
I cant think of a 'great' solution but this gives no conflicts
start:
ExpressionLoop
;
ExpressionLoop:
Expression
| ExpressionLoop ';' Expression
;
Expression:
rval
| rval '.' TYPENAME
| THIS //trick is moving this AWAY so it doesnt reduce
rval:
THIS '.' VARNAME
Alternative you can make it reduce later by adding more to the rule so it doesnt reduce as soon or by adding a token after or before to make it clear which path to take or fails (remember, it must know BEFORE reducing ANY path)
start:
ExpressionLoop
;
ExpressionLoop:
Expression
| ExpressionLoop ';' Expression
;
Expression:
rval
| rval '.' TYPENAME
rval:
THIS '#'
| THIS '.' VARNAME
%%
-edit- note if i want to do func param and type varname i cant because type according to the lexer func is a Var (which is A-Za-z09_) as well as type. param and varname are both var's as well so this will cause me a reduce/reduce conflict. You cant write this as what they are, only what they look like. So keep that in mind when writing. You'll have to write a token to differentiate the two or write it as one of the two but write additional logic in code (the part that is in { } on the right side of the rules) to check if it is a funcname or a type and handle both those case.