Gnu Bison shift / reduce conflicts in indentation-based grammar describing hierarchical expressions - parsing

Question

I've been not using Bison for a long time, so there is a chance I'm missing something simple here, however, I cannot figure out why the following grammar produces shift / reduce conflicts. I think the following grammar is not ambiguous. It's purpose is to parse expressions like:
a b
c d
e f
g h
as (in pseudo-AST):
App
(App a b)
(Seq
[ App
(App c d)
(Seq [App e f])
, (App g h)
]
)
The grammar:
%token <Token> VAR
%token <Token> EOL
%token <Token> INDENT_INC
%token <Token> INDENT_DEC
%token <AST> CONS
%token <AST> WILDCARD
%type <AST> expr
%type <AST> subExpr
%type <AST> block
%type <AST> tok
%start program
%%
program:
expr { result = $1; }
expr:
subExpr {$$=$1;}
| subExpr EOL INDENT_INC block { $$ = AST.app($1,$3); }
subExpr:
tok {$$=$1;}
| subExpr tok {$$ = AST.app($1,$2); }
block:
expr {$$=$1;}
| block EOL expr {$$=AST.seq($1,$3);} // causes error
tok:
VAR { $$ = AST.fromToken($1); }
%%
The error is just 2 shift/reduce conflicts. When debugging the parser, we can observe:
Grammar
0 $accept: program $end
1 program: expr
2 expr: subExpr
3 | subExpr EOL INDENT_INC block
4 subExpr: tok
5 | subExpr tok
6 block: expr
7 | block EOL expr
8 tok: VAR
[...]
State 4
2 expr: subExpr .
3 | subExpr . EOL INDENT_INC block
5 subExpr: subExpr . tok
VAR shift, and go to state 1
EOL shift, and go to state 7
EOL [reduce using rule 2 (expr)]
$default reduce using rule 2 (expr)
tok go to state 8
[...]
State 11
3 expr: subExpr EOL INDENT_INC block .
7 block: block . EOL expr
EOL shift, and go to state 12
EOL [reduce using rule 3 (expr)]
$default reduce using rule 3 (expr)
And to be honest, I'm not convinced where the ambiguity comes from. I'd be thankful for any help on how to remove the conflicts in such a grammar.

Answer 1

Your grammar does not use INDENT_DEC; without that, you cannot know where an indented block ends.
In effect, that's what those shift/reduce conflicts are telling you. Since the grammar doesn't see INDENT_DEC, it cannot distinguish between the EOL which separates two exprs in the same block and the EOL which terminates a block. Thus, an EOL is ambiguous (provided at least one INDENT_INC has been seen).
Here's a simple demonstration of ambiguity. The expression to parse is:
a EOL INDENT_INC b EOL INDENT_INC c EOL d
Here are two leftmost derivations, which differ in where d is nested (I condensed the subexpr ⇒ var ⇒ TOK path for simplicity):
# Here, d belongs to the outer subexpr (effectively, a single indent)
expr ⇒ subexpr EOL INDENT_INC block
⇒ TOK (a) EOL INDENT_INC block
⇒ TOK (a) EOL INDENT_INC block EOL expr
⇒ TOK (a) EOL INDENT_INC expr EOL expr
⇒ TOK (a) EOL INDENT_INC subexpr EOL INDENT_INC block EOL expr
⇒ TOK (a) EOL INDENT_INC subexpr EOL INDENT_INC expr EOL expr
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC subexpr EOL expr
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC TOK (c) EOL expr
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC TOK (c) EOL subexpr
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC TOK (c) EOL TOK (d)
# Here, d belongs to the inner subexpr (effectively two indents)
expr ⇒ subexpr EOL INDENT_INC block
⇒ TOK (a) EOL INDENT_INC block
⇒ TOK (a) EOL INDENT_INC expr
⇒ TOK (a) EOL INDENT_INC subexpr EOL INDENT_INC block
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC block
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC block EOL expr
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC expr EOL expr
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC subexpr EOL expr
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC TOK (c) EOL expr
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC TOK (c) EOL subexpr
⇒ TOK (a) EOL INDENT_INC TOK (b) EOL INDENT_INC TOK (c) EOL TOK (d)
So the grammar really is ambiguous. But the shift/reduce conflicts don't directly point at the ambiguity. They point at the problem of deciding whether or not to reduce the construct before the EOL without seeing the symbol following the EOL. This is the essence of the LR(1) restriction: Every reduction must be made before shifting the next symbol, so even if a grammar would be unambiguous if you could see far enough into the future it will still have shift/reduce conflicts if the reduction decision could go either way.