when looking for Information when and why to use Exceptions there are many people (also on this platform) making the point of not using exceptions when validating user-input because invalid input is not an exceptional thing to happen.
I now have the case where I have to parse a complex string of user input and map it to an Object-Tree basically, similar to a Parser.
Example in pseudo code:
input:
----
hello[5]
+
foo["ok"]
----
results in something like that:
class Hello {
int id = 5
}
class Add {}
class foo {
string name = 'ok'
}
Now in order to "validate" that input I have to parse it, having code that parses the input for validation and code to create the objects separately feels redundant.
Currently I'm using Exceptions while parsing single tokens to collect all Errors.
// one token is basically a single
try {
foreach (token in tokens) {
factory = getFactory(token) // throws ParseException
addObject(factory.create(token)) // throws ParseException
}
} catch (ParseException e) {
// e.g. "Foo Token expects value to be string"
addError(e)
}
is this bad use of exceptions?
An alternative would be to inject a validation class in every factory or mess around with return types (feels a bit dirty)
If exceptions work for your use case, go for it.
The usual problem with exceptions is that they don't let you fix things up and continue, which makes it hard to implement parser error recovery. You can't really fix up a bad input, and you probably shouldn't even in cases where you could, but error recovery lets you report more than one error from the same input, which is often considered convenient.
All of that depends on your needs and parsing strategy, so there's not a lot of information to go on here.
I need to parse a "token-level" language, i.e. the input is already tokenized with a semicolon as a delimiter. Sample input: A;B;A;D0;ASSIGN;X;. Here's also my grammar file.
I'd like to track location columns per-token. For the previous example, here's how I'd like to have columns defined:
Input: A;B;A;D0;ASSIGN;X;\n
Col: 1122334445555555666
So basically I'd like to increment column every time a semicolon is hit. I made a function that increments column count when semicolon is hit and for all actions I just set column in yylloc to my custom column count. However, with this approach I have to copypaste a function call to every action. Do you please know if there's any other cleaner way? Also there'll be no lexical errors in the input since it's autogenerated.
Edit: Nevermind, my solution actually doesn't work. So I'll be happy for any suggestions :)
%lex
%{
var delimit = (terminal) => { this.begin('delimit'); return terminal }
var columnInc = () => {
if (yy.lastLine === undefined) yy.lastLine = -1
if (yylloc.first_line !== yy.lastLine) {
yy.lastLine = yylloc.first_line
yy.columnCount = 0
}
yy.columnCount++
}
var setColumn = () => {
yylloc.first_column = yylloc.last_column = yy.columnCount
}
%}
%x delimit
%%
"ASSIGN" { return delimit('ASSIGN'); setColumn() }
"A" { return delimit('A'); setColumn() }
<delimit>{DELIMITER} { columnInc(); this.popState(); setColumn() }
\n { setColumn() }
...
There are a few ways to accomplish this in jison-gho. As you're looking to implement a token counter which is tracked by the parser, this invariably means we need to find a way to 'hook' into the code path where the lexer passes tokens to the parser.
Before we go look at a few implementations, a few thoughts that may help others who are facing similar, yet slightly different problems:
completely custom lexer for prepared token streams: as your input is a set of tokens already, one might consider using a custom lexer which would then just take the input stream as-is and do as little as possible while passing the tokens to the parser. This is doable in jison-gho and a fairly minimal example of such is demonstrated here:
https://github.com/GerHobbelt/jison/blob/0.6.1-215/examples/documentation--custom-lexer-ULcase.jison
while another way to integrate that same custom lexer is demonstrated here:
https://github.com/GerHobbelt/jison/blob/0.6.1-215/examples/documentation--custom-lexer-ULcase-alt.jison
or you might want to include the code from an external file via a %include "documentation--custom-lexer-ULcase.js" statement. Anyway, I digress.
Given your problem, depending on where that token stream comes from (who turns that into text? Is that outside your control as there's a huge overhead cost there as you're generating, then parsing a very long stream of text, while a custom lexer and some direct binary communications might reduce network or other costs there.
The bottom line is: if the token generator and everything up to this parse point is inside your control, I personally would favor a custom lexer and no text conversion what-so-ever for the intermediary channel. But in the end, that depends largely on your system requirements, context, etc. and is way outside the realm of this SO coding question.
augmenting the jison lexer: of course another approach could be to augment all (or a limited set of) lexer rules' action code, where you modify yytext to pass this data to the parser. This is the classic approach from the days of yacc/bison. Indeed, yytext doesn't have to be a string, but can be anything, e.g.
[a-z] %{
yytext = new DataInstance(
yytext, // the token string
yylloc, // the token location info
... // whatever you want/need...
);
return 'ID'; // the lexer token ID for this token
%}
For this problem, this is a lot of code duplication and thus a maintenance horror.
hooking into the flow between parser and lexer: this is new and facilitated by the jison-gho tool by pre_lex and post_lex callbacks. (The same mechanism is available around the parse() call so that you can initialize and postprocess a parser run in any way you want: pre_parse and post_parse are for that.
Here, since we want to count tokens, the simplest approach would be using the post_lex hook, which is only invoked when the lexer has completely parsed yet another token and passes it to the parser. In other words: post_lex is executed at the very end of the lex() call in the parser.
The documentation for these is included at the top of every generated parser/lexer JS source file, but then, of course, you need to know about that little nugget! ;-)
Here it is:
parser.lexer.options:
pre_lex: function()
optional: is invoked before the lexer is invoked to produce another token.
this refers to the Lexer object.
post_lex: function(token) { return token; }
optional: is invoked when the lexer has produced a token token;
this function can override the returned token value by returning another.
When it does not return any (truthy) value, the lexer will return
the original token.
this refers to the Lexer object.
Do note that options 1 and 3 are not available in vanilla jison, with one remark about option 1: jison does not accept a custom lexer as part of the jison parser/lexer spec file as demonstrated in the example links above. Of course, you can always go around and wrap the generated parser and thus inject a custom lexer and do other things.
Implementing the token counter using post_lex
Now how does it look in actual practice?
Solution 1: Let's do it nicely
We are going to 'abuse'/use (depending on your POV about riding on undocumented features) the yylloc info object and augment it with a counter member. We choose to do this so that we never risk interfering (or getting interference from) the default text/line-oriented yylloc position tracking system in the lexer and parser.
The undocumented bit here is the knowledge that all data members of any yylloc instance will be propagated by the default jison-gho location tracking&merging logic in the parser, hence when you tweak an yylloc instance in the lexer or parser action code, and if that yylloc instance is propagated to the output via merge or copy as the parser walks up the grammar tree, then your tweaks will be visible in the output.
Hooking into the lexer token output means we'll have to augment the lexer first, which we can easily do in the %% section before the /lex end-of-lexer-spec-marker:
// lexer extra code
var token_counter = 0;
lexer.post_lex = function (token) {
// hello world
++token_counter;
this.yylloc.counter = token_counter;
return token;
};
// extra helper so multiple parse() calls will restart counting tokens:
lexer.reset_token_counter = function () {
token_counter = 0;
};
where the magic bit is this statement: this.yylloc.counter = token_counter.
We hook a pre_lex callback into the flow by directly injecting it into the lexer definition via lexer.post_lex = function (){...}.
We could also have done this via the lexer options: lexer.options.post_lex = function ...
or via the parser-global yy instance: parser.yy.post_lex = function ... though those approaches would have meant we'ld be doing this in the parser definition code chunk or from the runtime which invokes the parser. These two slightly different approaches will not be demonstrated here.
Now all we have to do is complete this with a tiny bit of pre_parse code to ensure multiple parser.parse(input) invocations each will restart with the token counter reset to zero:
// extra helper: reset the token counter at the start of every parse() call:
parser.pre_parse = function (yy) {
yy.lexer.reset_token_counter();
};
Of course, that bit has to be added to the parser's final code block, after the %% in the grammar spec part of the jison file.
Full jison source file is available as a gist here.
How to compile and test:
# compile
jison --main so-q-58891186-2.jison
# run test code in main()
node so-q-58891186-2.js
Notes: I have 'faked' the AST construction code in your original source file so that one can easily diff the initial file with the one provided here. All that hack-it-to-make-it-work stuff is at the bottom part of the file.
Solution 2: Be a little nasty and re-use the yylloc.column location info and tracking
Instead of using the line info part of yylloc, I chose to use the column part instead, as to me that's about the same granularity level as a token sequence index. Doesn't matter which one you use, line or column, as long as you follow the same pattern.
When we do this right, we get the location tracking features of jison-gho added in for free, which is: column and line ranges for a grammar rule are automatically calculated from the individual token yylloc info in such a way that the first/last members of yylloc will show the first and last column, pardon, token index of the token sequence which is matched by the given grammar rule. This is the classic,merge jison-gho behaviour as mentioned in the --default-action CLI option:
--default-action
Specify the kind of default action that jison should include for every parser rule.
You can specify a mode for value handling ($$) and one for location
tracking (#$), separated by a comma, e.g.:
--default-action=ast,none
Supported value modes:
classic : generate a parser which includes the default
$$ = $1;
action for every rule.
ast : generate a parser which produces a simple AST-like
tree-of-arrays structure: every rule produces an array of
its production terms' values. Otherwise it is identical to
classic mode.
none : JISON will produce a slightly faster parser but then you are
solely responsible for propagating rule action $$ results.
The default rule value is still deterministic though as it
is set to undefined: $$ = undefined;
skip : same as none mode, except JISON does NOT INJECT a default
value action ANYWHERE, hence rule results are not
deterministic when you do not properly manage the $$ value
yourself!
Supported location modes:
merge : generate a parser which includes the default #$ = merged(#1..#n); location tracking action for every rule,
i.e. the rule's production 'location' is the range spanning its terms.
classic : same as merge mode.
ast : ditto.
none : JISON will produce a slightly faster parser but then you are solely responsible for propagating rule action #$ location results. The default rule location is still deterministic though, as it is set to undefined: #$ = undefined;
skip : same as "none" mode, except JISON does NOT INJECT a default location action ANYWHERE, hence rule location results are not deterministic when you do not properly manage the #$ value yourself!
Notes:
when you do specify a value default mode, but DO NOT specify a location value mode, the latter is assumed to be the same as the former.
Hence:
--default-action=ast
equals:
--default-action=ast,ast
when you do not specify an explicit default mode or only a "true"/"1" value, the default is assumed: classic,merge.
when you specify "false"/"0" as an explicit default mode, none,none is assumed. This produces the fastest deterministic parser.
Default setting: [classic,merge]
Now that we are going to 're-use' the fist_column and last_column members of yylloc instead of adding a new counter member, the magic bits that do the work remain nearly the same as in Solution 1:
augmenting the lexer in its %% section:
// lexer extra code
var token_counter = 0;
lexer.post_lex = function (token) {
++token_counter;
this.yylloc.first_column = token_counter;
this.yylloc.last_column = token_counter;
return token;
};
// extra helper so multiple parse() calls will restart counting tokens:
lexer.reset_token_counter = function () {
token_counter = 0;
};
Side Note: we 'abuse' the column part for tracking the token number; meanwhile the range member will still be usable to debug the raw text input as that one will track the positions within the raw input string.
Make sure to tweak both first_column and last_column so that the default location tracking 'merging' code in the generated parser can still do its job: that way we'll get to see which range of tokens constitute a particular grammar rule/element, just like
it were text columns.
Could've done the same with first_line/last_line, but I felt it more suitable to use the column part for this as it's at the same very low granularity level as 'token index'...
We hook a pre_lex callback into the flow by directly injecting it into the lexer definition via lexer.post_lex = function (){...}.
Same as Solution 1, now all we have to do is complete this with a tiny bit of pre_parse code to ensure multiple parser.parse(input) invocations each will restart with the token counter reset to zero:
// extra helper: reset the token counter at the start of every parse() call:
parser.pre_parse = function (yy) {
yy.lexer.reset_token_counter();
};
Of course, that bit has to be added to the parser's final code block, after the %% in the grammar spec part of the jison file.
Full jison source file is available as a gist here.
How to compile and test:
# compile
jison --main so-q-58891186-3.jison
# run test code in main()
node so-q-58891186-3.js
Aftermath / Observations about the solutions provided
Observe the test verification data at the end of both those jison files provided for how the token index shows up in the parser output:
Solution 1 (stripped, partial) output:
"type": "ProgramStmt",
"a1": [
{
"type": "ExprStmt",
"a1": {
"type": "AssignmentValueExpr",
"target": {
"type": "VariableRefExpr",
"a1": "ABA0",
"loc": {
"range": [
0,
8
],
"counter": 1
}
},
"source": {
"type": "VariableRefExpr",
"a1": "X",
"loc": {
"counter": 6
}
},
"loc": {
"counter": 1
}
},
"loc": {
"counter": 1
}
}
],
"loc": {
"counter": 1
}
Note here that the counter index is not really accurate for compound elements, i.e. elements which were constructed from multiple tokens matching one or more grammar rules: only the first token index is kept.
Solution 2 fares much better in that regard:
Solution 2 (stripped, partial) output:
"type": "ExprStmt",
"a1": {
"type": "AssignmentValueExpr",
"target": {
"type": "VariableRefExpr",
"a1": "ABA0",
"loc": {
"first_column": 1,
"last_column": 4,
}
},
"source": {
"type": "VariableRefExpr",
"a1": "X",
"loc": {
"first_column": 6,
"last_column": 6,
}
},
"loc": {
"first_column": 1,
"last_column": 6,
}
},
"loc": {
"first_column": 1,
"last_column": 7,
}
}
As you can see the first_column plus last_column members nicely track the set of tokens which constitute each part.
(Note that the counter increment code implied we start counting with ONE(1), not ZERO(0)!)
Parting thought
Given the input A;B;A;D0;ASSIGN;X;SEMICOLON; the current grammar parses this like ABA0 = X; and I wonder if this is what you really intend to get: constructing the identifier ABA0 like that seems a little odd to me.
Alas, that's not relevant to your question. It's just me encountering something quite out of the ordinary here, that's all. No matter.
Cheers and hope this long blurb is helpful to more of us. :-)
Source files:
original OP file as gist
solution 1 JISON file
solution 2 JISON file
current jison-gho release example grammars, including several which demo advanced features
I'm reading the source of polux's great parsers, and found there is a special isCommitted property which I can't understand:
class ParseResult<A> {
final bool isSuccess;
final bool isCommitted;
/// [:null:] if [:!isSuccess:]
final A value;
final String text;
final Position position;
final Expectations expectations;
// ...
}
You can see there is already a isSuccess to indicate the parse result is successful or not, why do we need a isCommitted? I tried to read related code, but still don't understand.
If you want to see the source, you can find it here.
The short answer is: don't worry about isCommited, it's for internal purposes only.
The long answer is: you can call commited on a paser, which means that once it has succeeded, you know for sure that it's pointless to backtrack (very much like Prolog's cut). For instance consider a grammar like this:
expr() => str('(') + rec(expr) str(')') ^ ...
| num()
Assume we parse the string "(...". Once we have recognized the parenthesis, we know for sure that if ... turns out not to be an expr, there is no need to rewind to the start of the string and try to parse a num, since a num will never start with a parenthesis anyway. We can fail early. This is done by marking ( as being a "commit point":
expr() => str('(').commited + rec(expr) str(')') ^ ...
| num()
This is an optimisation which should be used with great care because it breaks the modularity of parsers with respect to |. I personally never had to use it so far.
Whenever you call commited on a parser, it returns a new parser whose isCommited property is true. It is then used by | to decide whether to backtrack or not. This is what isCommited is used for. As an end user you should never have to care. I should probably make it private.
This feature is inspired by Polyparse's commit.
I have written parser_sub.mly and lexer_sub.mll which can parse a subroutine. A subroutine is a block of statement englobed by Sub and End Sub.
Actually, the raw file I would like to deal with contains a list of subroutines and some useless texts. Here is an example:
' a example file
Sub f1()
...
End Sub
haha
' hehe
Sub f2()
...
End Sub
So I need to write parser.mly and lexer.mll which can parse this file by ignoring all the comments (e.g. haha, ' hehe, etc.) and calling parser_sub.main, and returns a list of subroutines.
Could anyone tell me how to let the parser ignore all the useless sentences (sentences outside a Sub and End Sub)?
Here is a part of parser.mly I tried to write:
%{
open Syntax
%}
%start main
%type <Syntax.ev> main
%%
main:
subroutine_declaration* { $1 };
subroutine_declaration:
SUB name = subroutine_name LPAREN RPAREN EOS
body = procedure_body?
END SUB
{ { subroutine_name = name;
procedure_body_EOS_opt = body; } }
The rules and parsing for procedure_body are complex and are actually defined in parser_sub.mly and lexer_sub.mll, so how could I let parser.mly and lexer.mll do not repeat defining it, and just call parser_sub.main?
Maybe we can set some flag when we are inside subroutine:
sub_starts:
SUB { inside:=true };
sub_ends:
ENDSUB { inside:=false };
subroutine_declaration:
sub_starts name body sub_ends { ... }
And when this flag is not set you just skip any input?
If the stuff you want so skip can have any form (not necessarily valid tokens of your language), you pretty much have to solve this by hacking your lexer, as Kakadu suggests. This may be the easiest thing in any case.
If the filler (stuff to skip) consists of valid tokens, and you want to skip using a grammar rule, it seems to me the main problem is to define a nonterminal that matches any token other than END. This will be unpleasant to keep up to date, but seems possible.
Finally you have the problem that your end marker is two symbols, END SUB. You have to handle the case where you see END not followed by SUB. This is even trickier because SUB is your beginning marker also. Again, one way to simplify this would be to hack your lexer so that it treats END SUB as a single token. (Usually this is trickier than you'd expect, say if you want to allow comments between END and SUB.)
I've written a lexer and parser using scala.util.parsing.combinators.Parsers. I have a bug in at least one of my productions, but I have so many of them that it is difficult to eyeball them to determine the problem.
What I need is a log of every attempt my Parser makes to match the input with any production; logging all the Success and Failure objects when they are instantiated would be lovely. Unfortunately, the only way I can see to do this is to extend a lot of the basic classes provided by the library, then rewriting my massive parser to extend the new classes.
Is there an easy way to get this logging behavior?
You could use the log combinator to wrap productions of your grammar. Here's the definition in Parsers.scala:
def log[T](p: => Parser[T])(name: String): Parser[T] = Parser{ in =>
println("trying "+ name +" at "+ in)
val r = p(in)
println(name +" --> "+ r)
r
}
Otherwise, I think you should be able to override success and failure, but it would be quite uninformative, since you won't know what production called them.