Error: popping nterm - parsing

Question

I'm trying to understand the diagnostic messages given by Flex:
Entering state 5
Return for a new token:
Reading a token: Next token is token END_OF_FILE (4.0: )
Shifting token END_OF_FILE (4.0: )
Entering state 43
Reducing stack by rule 143 (line 331):
$1 = nterm syntax (0.0-17: )
$2 = nterm top_levels (0.18-4.0: )
$3 = token END_OF_FILE (4.0: )
-> $$ = nterm s (0.0-4.0: )
Stack now 0
Entering state 3
Return for a new token:
Reading a token: Next token is token END_OF_FILE (4.0: )
4/0: syntax error
Error: popping nterm s (0.0-4.0: )
Stack now 0
Cleanup: discarding lookahead token END_OF_FILE (4.0: )
Stack now 0
I cannot understand why / what is it trying to do with EOF token. Below are the Flex rules:
<<EOF>> { return END_OF_FILE; }
And the Bison rules:
top_level : message
| enum
| service
| import { $$ = Py_None; }
| package { $$ = Py_None; }
| option_def { $$ = Py_None; }
| ';' { $$ = Py_None; } ;
top_levels : %empty { $$ = py_list(Py_None); }
| top_levels top_level { $$ = py_append($1, $2); } ;
s : syntax top_levels END_OF_FILE { $$ = $2; } ;
And the output file generated by Bison:
State 3
0 $accept: s . $end
$end shift, and go to state 6
State 5
142 top_levels: top_levels . top_level
143 s: syntax top_levels . END_OF_FILE
BOOL shift, and go to state 9
... bunch of similar rules
END_OF_FILE shift, and go to state 43
';' shift, and go to state 44
import go to state 45
... bunch of similar rules
top_level go to state 55
State 6
0 $accept: s $end .
$default accept
I have no idea what's going on. Why does it report reading EOF token twice? What was exactly the problem with popping s? To me it seems like it actually accepted the whole thing, and then decided to reject it because it red the token second time... but the whole reporting is very confusing.

Answer 1

1. The problem
Don't do this:
<<EOF>> { return END_OF_FILE; }
Yacc/bison parsers augment grammars with an internal rule which produces the start symbol followed by an internal eof token called $end, whose token number is 0. (You can see this rule in states 3 and 6.) That is the only accepting rule in the grammar.
By default, (f)lex scanners return 0 when EOF is detected. So that all Just Works.
When you try to send a different token on EOF, you are attempting to defeat this mechanism, but it won't work because the start symbol is not an accepting rule. After the start symbil is reduced, the parser tries to reduce the $accept rule, so it asks the scanner for another token. But the scanner has already hit EOF. In most cases, the scanner will execute the <<EOF>> action again (although this is not guaranteed), but that's not going to produce the $end token it needs. So you get a syntax error.
2. The underlying problem (maybe)
Normally, people try this in order to create a user action which runs when the input is accepted, typically in order to return the result of the parse to yyparse's caller through an "out" parameter. Trying to explicitly recognize an EOF token (or even the $end token) in the start production cannot work, but there is a much simpler solution: an extra unit rule:
%start return
%%
return: s { *out = $1; }
s: syntax top_levels { $$ = $2; }
Note that you could also do this without top_levels:
%start return
%%
return: { *out = $1; }
s: syntax { $$ = py_list(Py_None); }
| s top_level { $$ = py_append($1, $2); }
An alternative is to use the special YYACCEPT action macro in the action for the start rule. However, I believe the standard solution outlined above is simpler because it doesn't require anything from the scanner.
3. The trace output
Error: popping nterm s (0.0-4.0: )
Means:
A syntax error was detected.
As part of error recovery, the parser popped the non-terminal s from the stack.
That non-terminal's source location extends from 0.0 to 4.0 (line . column)
If s (or its semantic type) had had a registered destructor, that would have run at step 2. You probably will want to register destructor for syntactic types which reference Python values in order to decrement their reference counts so that you don't leak memory on syntax errors. But perhaps I'm wrong about that.
Also, you could register a %printer for the syntactic value, in which case it would have been printed after the colon.