I have defined a grammar and now I'm implementing a parser for it.
The program should start with the keyword main followed by an opening curly bracket, followed in turn by a (possibly empty) sequence of statements, and terminated by a closing curly bracket. My questions is how to define the program in the parser? I have tried several different ways, including this, but it doesn't seem to be correct when I test.
public void program() {
//Program -> MAIN LCBR Statement* RCBR
eat("MAIN");
eat("LCBR");
while (lex.token().type != "RCBR") {
statement();
}
}
Any suggestions would be appreciated!
I am trying to add instrumentation (e.g. logging some information) to methods in a Java file. I am using the following Rascal code which seems to work mostly:
import ParseTree;
import lang::java::\syntax::Java15;
// .. more imports
// project is a loc
M3 model = createM3FromEclipseProject(project);
set[loc] projectFiles = { file | file <- files(model)} ;
for (pFile <- projectFiles) {
CompilationUnit cunit = parse(#CompilationUnit, pFile);
cUnitNew = visit(cunit) {
case (MethodBody) `{<BlockStm* post>}`
=> (MethodBody) `{
'System.out.println(new Throwable().getStackTrace()[0]);
'<BlockStm* post>
'}`
}
writeFile(pFile, cUnitNew);
}
I am running into two issues regarding whitespace, which might be unrelated.
The line of code that I am inserting does not preserve whitespace that was there previously. If there was a tab character, it will now be removed. The same is true for the line directly following the line I am inserting and the closing brace. How can I 'capture' whitespace in my pattern?
Example before transforming (all lines start with a tab character, line 2 and 3 with two):
void beforeFirst() throws Exception {
rowIdx = -1;
rowSource.beforeFirst();
}
Example after transforming:
void beforeFirst() throws Exception {
System.out.println(new Throwable().getStackTrace()[0]);
rowIdx = -1;
rowSource.beforeFirst();
}
An additional issue regarding whitespace; if a file ends on a newline character, the parse function will throw a ParseError without further details. Removing this newline from the original source will fix the issue, but I'd rather not 'manually' have to fix code before parsing. How can I circumvent this issue?
Alas, capturing whitespace with a concrete pattern is not a feature of the current version of Rascal. We used to have it, but now it's back on the TODO list. I can point you to papers about the topic if you are interested. So for now you have to deal with this "damage" later.
You could write a Tree to Tree transformation on the generic level (see ParseTree.rsc), to fix indentation issues in a parse tree after your transformation, or to re-insert the comments that you lost. This is about matching the Tree data-type and appl constructors. The Tree format is a form of reflection on the parse trees of Rascal that allow any kind of transformation, including whitespace and comments.
The parse error you talked about is caused by not using the start non-terminal. If you use parse(#start[CompilationUnit], ...) then whitespace and comments before and after the CompilationUnit are accepted.
I need to parse a "token-level" language, i.e. the input is already tokenized with a semicolon as a delimiter. Sample input: A;B;A;D0;ASSIGN;X;. Here's also my grammar file.
I'd like to track location columns per-token. For the previous example, here's how I'd like to have columns defined:
Input: A;B;A;D0;ASSIGN;X;\n
Col: 1122334445555555666
So basically I'd like to increment column every time a semicolon is hit. I made a function that increments column count when semicolon is hit and for all actions I just set column in yylloc to my custom column count. However, with this approach I have to copypaste a function call to every action. Do you please know if there's any other cleaner way? Also there'll be no lexical errors in the input since it's autogenerated.
Edit: Nevermind, my solution actually doesn't work. So I'll be happy for any suggestions :)
%lex
%{
var delimit = (terminal) => { this.begin('delimit'); return terminal }
var columnInc = () => {
if (yy.lastLine === undefined) yy.lastLine = -1
if (yylloc.first_line !== yy.lastLine) {
yy.lastLine = yylloc.first_line
yy.columnCount = 0
}
yy.columnCount++
}
var setColumn = () => {
yylloc.first_column = yylloc.last_column = yy.columnCount
}
%}
%x delimit
%%
"ASSIGN" { return delimit('ASSIGN'); setColumn() }
"A" { return delimit('A'); setColumn() }
<delimit>{DELIMITER} { columnInc(); this.popState(); setColumn() }
\n { setColumn() }
...
There are a few ways to accomplish this in jison-gho. As you're looking to implement a token counter which is tracked by the parser, this invariably means we need to find a way to 'hook' into the code path where the lexer passes tokens to the parser.
Before we go look at a few implementations, a few thoughts that may help others who are facing similar, yet slightly different problems:
completely custom lexer for prepared token streams: as your input is a set of tokens already, one might consider using a custom lexer which would then just take the input stream as-is and do as little as possible while passing the tokens to the parser. This is doable in jison-gho and a fairly minimal example of such is demonstrated here:
https://github.com/GerHobbelt/jison/blob/0.6.1-215/examples/documentation--custom-lexer-ULcase.jison
while another way to integrate that same custom lexer is demonstrated here:
https://github.com/GerHobbelt/jison/blob/0.6.1-215/examples/documentation--custom-lexer-ULcase-alt.jison
or you might want to include the code from an external file via a %include "documentation--custom-lexer-ULcase.js" statement. Anyway, I digress.
Given your problem, depending on where that token stream comes from (who turns that into text? Is that outside your control as there's a huge overhead cost there as you're generating, then parsing a very long stream of text, while a custom lexer and some direct binary communications might reduce network or other costs there.
The bottom line is: if the token generator and everything up to this parse point is inside your control, I personally would favor a custom lexer and no text conversion what-so-ever for the intermediary channel. But in the end, that depends largely on your system requirements, context, etc. and is way outside the realm of this SO coding question.
augmenting the jison lexer: of course another approach could be to augment all (or a limited set of) lexer rules' action code, where you modify yytext to pass this data to the parser. This is the classic approach from the days of yacc/bison. Indeed, yytext doesn't have to be a string, but can be anything, e.g.
[a-z] %{
yytext = new DataInstance(
yytext, // the token string
yylloc, // the token location info
... // whatever you want/need...
);
return 'ID'; // the lexer token ID for this token
%}
For this problem, this is a lot of code duplication and thus a maintenance horror.
hooking into the flow between parser and lexer: this is new and facilitated by the jison-gho tool by pre_lex and post_lex callbacks. (The same mechanism is available around the parse() call so that you can initialize and postprocess a parser run in any way you want: pre_parse and post_parse are for that.
Here, since we want to count tokens, the simplest approach would be using the post_lex hook, which is only invoked when the lexer has completely parsed yet another token and passes it to the parser. In other words: post_lex is executed at the very end of the lex() call in the parser.
The documentation for these is included at the top of every generated parser/lexer JS source file, but then, of course, you need to know about that little nugget! ;-)
Here it is:
parser.lexer.options:
pre_lex: function()
optional: is invoked before the lexer is invoked to produce another token.
this refers to the Lexer object.
post_lex: function(token) { return token; }
optional: is invoked when the lexer has produced a token token;
this function can override the returned token value by returning another.
When it does not return any (truthy) value, the lexer will return
the original token.
this refers to the Lexer object.
Do note that options 1 and 3 are not available in vanilla jison, with one remark about option 1: jison does not accept a custom lexer as part of the jison parser/lexer spec file as demonstrated in the example links above. Of course, you can always go around and wrap the generated parser and thus inject a custom lexer and do other things.
Implementing the token counter using post_lex
Now how does it look in actual practice?
Solution 1: Let's do it nicely
We are going to 'abuse'/use (depending on your POV about riding on undocumented features) the yylloc info object and augment it with a counter member. We choose to do this so that we never risk interfering (or getting interference from) the default text/line-oriented yylloc position tracking system in the lexer and parser.
The undocumented bit here is the knowledge that all data members of any yylloc instance will be propagated by the default jison-gho location tracking&merging logic in the parser, hence when you tweak an yylloc instance in the lexer or parser action code, and if that yylloc instance is propagated to the output via merge or copy as the parser walks up the grammar tree, then your tweaks will be visible in the output.
Hooking into the lexer token output means we'll have to augment the lexer first, which we can easily do in the %% section before the /lex end-of-lexer-spec-marker:
// lexer extra code
var token_counter = 0;
lexer.post_lex = function (token) {
// hello world
++token_counter;
this.yylloc.counter = token_counter;
return token;
};
// extra helper so multiple parse() calls will restart counting tokens:
lexer.reset_token_counter = function () {
token_counter = 0;
};
where the magic bit is this statement: this.yylloc.counter = token_counter.
We hook a pre_lex callback into the flow by directly injecting it into the lexer definition via lexer.post_lex = function (){...}.
We could also have done this via the lexer options: lexer.options.post_lex = function ...
or via the parser-global yy instance: parser.yy.post_lex = function ... though those approaches would have meant we'ld be doing this in the parser definition code chunk or from the runtime which invokes the parser. These two slightly different approaches will not be demonstrated here.
Now all we have to do is complete this with a tiny bit of pre_parse code to ensure multiple parser.parse(input) invocations each will restart with the token counter reset to zero:
// extra helper: reset the token counter at the start of every parse() call:
parser.pre_parse = function (yy) {
yy.lexer.reset_token_counter();
};
Of course, that bit has to be added to the parser's final code block, after the %% in the grammar spec part of the jison file.
Full jison source file is available as a gist here.
How to compile and test:
# compile
jison --main so-q-58891186-2.jison
# run test code in main()
node so-q-58891186-2.js
Notes: I have 'faked' the AST construction code in your original source file so that one can easily diff the initial file with the one provided here. All that hack-it-to-make-it-work stuff is at the bottom part of the file.
Solution 2: Be a little nasty and re-use the yylloc.column location info and tracking
Instead of using the line info part of yylloc, I chose to use the column part instead, as to me that's about the same granularity level as a token sequence index. Doesn't matter which one you use, line or column, as long as you follow the same pattern.
When we do this right, we get the location tracking features of jison-gho added in for free, which is: column and line ranges for a grammar rule are automatically calculated from the individual token yylloc info in such a way that the first/last members of yylloc will show the first and last column, pardon, token index of the token sequence which is matched by the given grammar rule. This is the classic,merge jison-gho behaviour as mentioned in the --default-action CLI option:
--default-action
Specify the kind of default action that jison should include for every parser rule.
You can specify a mode for value handling ($$) and one for location
tracking (#$), separated by a comma, e.g.:
--default-action=ast,none
Supported value modes:
classic : generate a parser which includes the default
$$ = $1;
action for every rule.
ast : generate a parser which produces a simple AST-like
tree-of-arrays structure: every rule produces an array of
its production terms' values. Otherwise it is identical to
classic mode.
none : JISON will produce a slightly faster parser but then you are
solely responsible for propagating rule action $$ results.
The default rule value is still deterministic though as it
is set to undefined: $$ = undefined;
skip : same as none mode, except JISON does NOT INJECT a default
value action ANYWHERE, hence rule results are not
deterministic when you do not properly manage the $$ value
yourself!
Supported location modes:
merge : generate a parser which includes the default #$ = merged(#1..#n); location tracking action for every rule,
i.e. the rule's production 'location' is the range spanning its terms.
classic : same as merge mode.
ast : ditto.
none : JISON will produce a slightly faster parser but then you are solely responsible for propagating rule action #$ location results. The default rule location is still deterministic though, as it is set to undefined: #$ = undefined;
skip : same as "none" mode, except JISON does NOT INJECT a default location action ANYWHERE, hence rule location results are not deterministic when you do not properly manage the #$ value yourself!
Notes:
when you do specify a value default mode, but DO NOT specify a location value mode, the latter is assumed to be the same as the former.
Hence:
--default-action=ast
equals:
--default-action=ast,ast
when you do not specify an explicit default mode or only a "true"/"1" value, the default is assumed: classic,merge.
when you specify "false"/"0" as an explicit default mode, none,none is assumed. This produces the fastest deterministic parser.
Default setting: [classic,merge]
Now that we are going to 're-use' the fist_column and last_column members of yylloc instead of adding a new counter member, the magic bits that do the work remain nearly the same as in Solution 1:
augmenting the lexer in its %% section:
// lexer extra code
var token_counter = 0;
lexer.post_lex = function (token) {
++token_counter;
this.yylloc.first_column = token_counter;
this.yylloc.last_column = token_counter;
return token;
};
// extra helper so multiple parse() calls will restart counting tokens:
lexer.reset_token_counter = function () {
token_counter = 0;
};
Side Note: we 'abuse' the column part for tracking the token number; meanwhile the range member will still be usable to debug the raw text input as that one will track the positions within the raw input string.
Make sure to tweak both first_column and last_column so that the default location tracking 'merging' code in the generated parser can still do its job: that way we'll get to see which range of tokens constitute a particular grammar rule/element, just like
it were text columns.
Could've done the same with first_line/last_line, but I felt it more suitable to use the column part for this as it's at the same very low granularity level as 'token index'...
We hook a pre_lex callback into the flow by directly injecting it into the lexer definition via lexer.post_lex = function (){...}.
Same as Solution 1, now all we have to do is complete this with a tiny bit of pre_parse code to ensure multiple parser.parse(input) invocations each will restart with the token counter reset to zero:
// extra helper: reset the token counter at the start of every parse() call:
parser.pre_parse = function (yy) {
yy.lexer.reset_token_counter();
};
Of course, that bit has to be added to the parser's final code block, after the %% in the grammar spec part of the jison file.
Full jison source file is available as a gist here.
How to compile and test:
# compile
jison --main so-q-58891186-3.jison
# run test code in main()
node so-q-58891186-3.js
Aftermath / Observations about the solutions provided
Observe the test verification data at the end of both those jison files provided for how the token index shows up in the parser output:
Solution 1 (stripped, partial) output:
"type": "ProgramStmt",
"a1": [
{
"type": "ExprStmt",
"a1": {
"type": "AssignmentValueExpr",
"target": {
"type": "VariableRefExpr",
"a1": "ABA0",
"loc": {
"range": [
0,
8
],
"counter": 1
}
},
"source": {
"type": "VariableRefExpr",
"a1": "X",
"loc": {
"counter": 6
}
},
"loc": {
"counter": 1
}
},
"loc": {
"counter": 1
}
}
],
"loc": {
"counter": 1
}
Note here that the counter index is not really accurate for compound elements, i.e. elements which were constructed from multiple tokens matching one or more grammar rules: only the first token index is kept.
Solution 2 fares much better in that regard:
Solution 2 (stripped, partial) output:
"type": "ExprStmt",
"a1": {
"type": "AssignmentValueExpr",
"target": {
"type": "VariableRefExpr",
"a1": "ABA0",
"loc": {
"first_column": 1,
"last_column": 4,
}
},
"source": {
"type": "VariableRefExpr",
"a1": "X",
"loc": {
"first_column": 6,
"last_column": 6,
}
},
"loc": {
"first_column": 1,
"last_column": 6,
}
},
"loc": {
"first_column": 1,
"last_column": 7,
}
}
As you can see the first_column plus last_column members nicely track the set of tokens which constitute each part.
(Note that the counter increment code implied we start counting with ONE(1), not ZERO(0)!)
Parting thought
Given the input A;B;A;D0;ASSIGN;X;SEMICOLON; the current grammar parses this like ABA0 = X; and I wonder if this is what you really intend to get: constructing the identifier ABA0 like that seems a little odd to me.
Alas, that's not relevant to your question. It's just me encountering something quite out of the ordinary here, that's all. No matter.
Cheers and hope this long blurb is helpful to more of us. :-)
Source files:
original OP file as gist
solution 1 JISON file
solution 2 JISON file
current jison-gho release example grammars, including several which demo advanced features
I'm working with a Java 8 grammar and I want to find occurrences of a method invocation, more specifically it.hasNext(), when it is an Iterator.
This works:
visit(unit) {
case (MethodInvocation)`it . <TypeArguments? ta> hasNext()`: {
println("found");
}
}
Ideally I would like to match with any identifier, not just it.
So I tried using String interpolation, which compiles but doesn't match:
str iteratorId = "it";
visit(unit) {
case (MethodInvocation)`$iteratorId$ . <TypeArguments? ta> hasNext()`: {
println("achei");
}
}
I also tried several other ways, including pattern variable uses (as seen in the docs) but I can't get this to work.
Is this kind of matching possible in rascal? If yes, how can it be done?
The answer specifically depends on the grammar you are using, which I did not look up, but in general in concrete syntax fragments this notation is used for placeholders: <NonTerminal variableName>
So your pattern should look something like the following:
str iteratorId = "it";
visit(unit) {
case (MethodInvocation)`<MethodName name>.<TypeArguments? ta>hasNext()`:
if (iteratorId == "<name>") println("bingo!");
}
That is assuming that MethodName is indeed a non-terminal in your Java8 grammar and part of the syntax rule for method invocations.
I'm using ANTLR4 to create a parse tree for my grammar, what I want to do is modify certain nodes in the tree. This will include removing certain nodes and inserting new ones. The purpose behind this is optimization for the language I am writing. I have yet to find a solution to this problem. What would be the best way to go about this?
While there is currently no real support or tools for tree rewriting, it is very possible to do. It's not even that painful.
The ParseTreeListener or your MyBaseListener can be used with a ParseTreeWalker to walk your parse tree.
From here, you can remove nodes with ParserRuleContext.removeLastChild(), however when doing this, you have to watch out for ParseTreeWalker.walk:
public void walk(ParseTreeListener listener, ParseTree t) {
if ( t instanceof ErrorNode) {
listener.visitErrorNode((ErrorNode)t);
return;
}
else if ( t instanceof TerminalNode) {
listener.visitTerminal((TerminalNode)t);
return;
}
RuleNode r = (RuleNode)t;
enterRule(listener, r);
int n = r.getChildCount();
for (int i = 0; i<n; i++) {
walk(listener, r.getChild(i));
}
exitRule(listener, r);
}
You must replace removed nodes with something if the walker has visited parents of those nodes, I usually pick empty ParseRuleContext objects (this is because of the cached value of n in the method above). This prevents the ParseTreeWalker from throwing a NPE.
When adding nodes, make sure to set the mutable parent on the ParseRuleContext to the new parent. Also, because of the cached n in the method above, a good strategy is to detect where the changes need to be before you hit where you want your changes to go in the walk, so the ParseTreeWalker will walk over them in the same pass (other wise you might need multiple passes...)
Your pseudo code should look like this:
public void enterRewriteTarget(#NotNull MyParser.RewriteTargetContext ctx){
if(shouldRewrite(ctx)){
ArrayList<ParseTree> nodesReplaced = replaceNodes(ctx);
addChildTo(ctx, createNewParentFor(nodesReplaced));
}
}
I've used this method to write a transpiler that compiled a synchronous internal language into asynchronous javascript. It was pretty painful.
Another approach would be to write a ParseTreeVisitor that converts the tree back to a string. (This can be trivial in some cases, because you are only calling TerminalNode.getText() and concatenate in aggregateResult(..).)
You then add the modifications to this visitor so that the resulting string representation contains the modifications you try to achieve.
Then parse the string and you get a parse tree with the desired modifications.
This is certainly hackish in some ways, since you parse the string twice. On the other hand the solution does not rely on antlr implementation details.
I needed something similar for simple transformations. I ended up using a ParseTreeWalker and a custom ...BaseListener where I overwrote the enter... methods. Inside this method the ParserRuleContext.children is available and can be manipulated.
class MyListener extends ...BaseListener {
#Override
public void enter...(...Context ctx) {
super.enter...(ctx);
ctx.children.add(...);
}
}
new ParseTreeWalker().walk(new MyListener(), parseTree);