I have defined a grammar and now I'm implementing a parser for it.
The program should start with the keyword main followed by an opening curly bracket, followed in turn by a (possibly empty) sequence of statements, and terminated by a closing curly bracket. My questions is how to define the program in the parser? I have tried several different ways, including this, but it doesn't seem to be correct when I test.
public void program() {
//Program -> MAIN LCBR Statement* RCBR
eat("MAIN");
eat("LCBR");
while (lex.token().type != "RCBR") {
statement();
}
}
Any suggestions would be appreciated!
I'm trying to re-create Tijs' CurryOn16 example "TrafoFields" scraping the code from the video, but using the Java18.rsc grammar instead of his Java15.rsc. I've parsed the Example.java successfully in the repl, like he did in the video, yielding a var pt. I then try to do the transformation with trafoFields(pt). The response I get is:
|project://Rascal-Test/src/TrafoFields.rsc|(235,142,<12,9>,<16,11>): Syntax error: concrete syntax fragment
My TrafoFields.rsc looks like this:
module TrafoFields
import lang::java::\syntax::Java18;
/**
* - Make public fields private
* - add getters and setters
*/
start[CompilationUnit] trafoFields(start[CompilationUnit] cu) {
return innermost visit (cu) {
case (ClassBody)`{
' <ClassBodyDeclaration* cs1>
' public <Type t> <ID f>;
' <ClassBodyDeclaration* cs2>
'}`
=> (ClassBody)`{
' <ClassBodyDeclaration* cs1>
' private <Type t> <ID f>;
' public void <ID setter>(<Type t> x) {
' this.<ID f> = x;
' }
' public <Type t> <ID getter>() {
' return this.<ID f>;
' }
' <ClassBodyDeclaration* cs2>
'}`
when
ID setter := [ID]"set<f>",
ID getter := [ID]"get<f>"
}
}
The only deviation from Tijs' code is that I've changed ClassBodyDec* to ClassBodyDeclaration*, as the grammar has this as a non-terminal. Any hint what else could be wrong?
UPDATE
More non-terminal re-writing adapting to Java18 grammar:
Id => ID
Ah yes, that is the Achilles-heal of concrete syntax usability; parse errors.
Note that a generalized parser (such as GLL which Rascal uses), simulates "unlimited lookahead" and so a parse error may be reported a few characters or even a few lines after the actual cause (but never before!). So shortening the example (delta debugging) will help localize the cause.
My way-of-life in this is:
First replace all pattern holes by concrete Java snippets. I know Java, so I should be able to write a correct fragment that would have matched the holes.
If there is still a parse error, now you check the top-non-terminal. Is it the one you needed? also make sure there is no extra whitespace before the start and after the end of the fragment inside the backquotes. Still a parse error? Write a shorter fragment first for a sub-nonterminal first.
Parse error solved? this means one of the pattern holes was not syntactically correct. The type of the hole is leading here, it should be one of the non-terminals used the grammar literally, and of course at the right spot in the fragment. Add the holes back in one-by-one until you hit the error again. Then you know the cause and probably also the fix.
I need to parse a "token-level" language, i.e. the input is already tokenized with a semicolon as a delimiter. Sample input: A;B;A;D0;ASSIGN;X;. Here's also my grammar file.
I'd like to track location columns per-token. For the previous example, here's how I'd like to have columns defined:
Input: A;B;A;D0;ASSIGN;X;\n
Col: 1122334445555555666
So basically I'd like to increment column every time a semicolon is hit. I made a function that increments column count when semicolon is hit and for all actions I just set column in yylloc to my custom column count. However, with this approach I have to copypaste a function call to every action. Do you please know if there's any other cleaner way? Also there'll be no lexical errors in the input since it's autogenerated.
Edit: Nevermind, my solution actually doesn't work. So I'll be happy for any suggestions :)
%lex
%{
var delimit = (terminal) => { this.begin('delimit'); return terminal }
var columnInc = () => {
if (yy.lastLine === undefined) yy.lastLine = -1
if (yylloc.first_line !== yy.lastLine) {
yy.lastLine = yylloc.first_line
yy.columnCount = 0
}
yy.columnCount++
}
var setColumn = () => {
yylloc.first_column = yylloc.last_column = yy.columnCount
}
%}
%x delimit
%%
"ASSIGN" { return delimit('ASSIGN'); setColumn() }
"A" { return delimit('A'); setColumn() }
<delimit>{DELIMITER} { columnInc(); this.popState(); setColumn() }
\n { setColumn() }
...
There are a few ways to accomplish this in jison-gho. As you're looking to implement a token counter which is tracked by the parser, this invariably means we need to find a way to 'hook' into the code path where the lexer passes tokens to the parser.
Before we go look at a few implementations, a few thoughts that may help others who are facing similar, yet slightly different problems:
completely custom lexer for prepared token streams: as your input is a set of tokens already, one might consider using a custom lexer which would then just take the input stream as-is and do as little as possible while passing the tokens to the parser. This is doable in jison-gho and a fairly minimal example of such is demonstrated here:
https://github.com/GerHobbelt/jison/blob/0.6.1-215/examples/documentation--custom-lexer-ULcase.jison
while another way to integrate that same custom lexer is demonstrated here:
https://github.com/GerHobbelt/jison/blob/0.6.1-215/examples/documentation--custom-lexer-ULcase-alt.jison
or you might want to include the code from an external file via a %include "documentation--custom-lexer-ULcase.js" statement. Anyway, I digress.
Given your problem, depending on where that token stream comes from (who turns that into text? Is that outside your control as there's a huge overhead cost there as you're generating, then parsing a very long stream of text, while a custom lexer and some direct binary communications might reduce network or other costs there.
The bottom line is: if the token generator and everything up to this parse point is inside your control, I personally would favor a custom lexer and no text conversion what-so-ever for the intermediary channel. But in the end, that depends largely on your system requirements, context, etc. and is way outside the realm of this SO coding question.
augmenting the jison lexer: of course another approach could be to augment all (or a limited set of) lexer rules' action code, where you modify yytext to pass this data to the parser. This is the classic approach from the days of yacc/bison. Indeed, yytext doesn't have to be a string, but can be anything, e.g.
[a-z] %{
yytext = new DataInstance(
yytext, // the token string
yylloc, // the token location info
... // whatever you want/need...
);
return 'ID'; // the lexer token ID for this token
%}
For this problem, this is a lot of code duplication and thus a maintenance horror.
hooking into the flow between parser and lexer: this is new and facilitated by the jison-gho tool by pre_lex and post_lex callbacks. (The same mechanism is available around the parse() call so that you can initialize and postprocess a parser run in any way you want: pre_parse and post_parse are for that.
Here, since we want to count tokens, the simplest approach would be using the post_lex hook, which is only invoked when the lexer has completely parsed yet another token and passes it to the parser. In other words: post_lex is executed at the very end of the lex() call in the parser.
The documentation for these is included at the top of every generated parser/lexer JS source file, but then, of course, you need to know about that little nugget! ;-)
Here it is:
parser.lexer.options:
pre_lex: function()
optional: is invoked before the lexer is invoked to produce another token.
this refers to the Lexer object.
post_lex: function(token) { return token; }
optional: is invoked when the lexer has produced a token token;
this function can override the returned token value by returning another.
When it does not return any (truthy) value, the lexer will return
the original token.
this refers to the Lexer object.
Do note that options 1 and 3 are not available in vanilla jison, with one remark about option 1: jison does not accept a custom lexer as part of the jison parser/lexer spec file as demonstrated in the example links above. Of course, you can always go around and wrap the generated parser and thus inject a custom lexer and do other things.
Implementing the token counter using post_lex
Now how does it look in actual practice?
Solution 1: Let's do it nicely
We are going to 'abuse'/use (depending on your POV about riding on undocumented features) the yylloc info object and augment it with a counter member. We choose to do this so that we never risk interfering (or getting interference from) the default text/line-oriented yylloc position tracking system in the lexer and parser.
The undocumented bit here is the knowledge that all data members of any yylloc instance will be propagated by the default jison-gho location tracking&merging logic in the parser, hence when you tweak an yylloc instance in the lexer or parser action code, and if that yylloc instance is propagated to the output via merge or copy as the parser walks up the grammar tree, then your tweaks will be visible in the output.
Hooking into the lexer token output means we'll have to augment the lexer first, which we can easily do in the %% section before the /lex end-of-lexer-spec-marker:
// lexer extra code
var token_counter = 0;
lexer.post_lex = function (token) {
// hello world
++token_counter;
this.yylloc.counter = token_counter;
return token;
};
// extra helper so multiple parse() calls will restart counting tokens:
lexer.reset_token_counter = function () {
token_counter = 0;
};
where the magic bit is this statement: this.yylloc.counter = token_counter.
We hook a pre_lex callback into the flow by directly injecting it into the lexer definition via lexer.post_lex = function (){...}.
We could also have done this via the lexer options: lexer.options.post_lex = function ...
or via the parser-global yy instance: parser.yy.post_lex = function ... though those approaches would have meant we'ld be doing this in the parser definition code chunk or from the runtime which invokes the parser. These two slightly different approaches will not be demonstrated here.
Now all we have to do is complete this with a tiny bit of pre_parse code to ensure multiple parser.parse(input) invocations each will restart with the token counter reset to zero:
// extra helper: reset the token counter at the start of every parse() call:
parser.pre_parse = function (yy) {
yy.lexer.reset_token_counter();
};
Of course, that bit has to be added to the parser's final code block, after the %% in the grammar spec part of the jison file.
Full jison source file is available as a gist here.
How to compile and test:
# compile
jison --main so-q-58891186-2.jison
# run test code in main()
node so-q-58891186-2.js
Notes: I have 'faked' the AST construction code in your original source file so that one can easily diff the initial file with the one provided here. All that hack-it-to-make-it-work stuff is at the bottom part of the file.
Solution 2: Be a little nasty and re-use the yylloc.column location info and tracking
Instead of using the line info part of yylloc, I chose to use the column part instead, as to me that's about the same granularity level as a token sequence index. Doesn't matter which one you use, line or column, as long as you follow the same pattern.
When we do this right, we get the location tracking features of jison-gho added in for free, which is: column and line ranges for a grammar rule are automatically calculated from the individual token yylloc info in such a way that the first/last members of yylloc will show the first and last column, pardon, token index of the token sequence which is matched by the given grammar rule. This is the classic,merge jison-gho behaviour as mentioned in the --default-action CLI option:
--default-action
Specify the kind of default action that jison should include for every parser rule.
You can specify a mode for value handling ($$) and one for location
tracking (#$), separated by a comma, e.g.:
--default-action=ast,none
Supported value modes:
classic : generate a parser which includes the default
$$ = $1;
action for every rule.
ast : generate a parser which produces a simple AST-like
tree-of-arrays structure: every rule produces an array of
its production terms' values. Otherwise it is identical to
classic mode.
none : JISON will produce a slightly faster parser but then you are
solely responsible for propagating rule action $$ results.
The default rule value is still deterministic though as it
is set to undefined: $$ = undefined;
skip : same as none mode, except JISON does NOT INJECT a default
value action ANYWHERE, hence rule results are not
deterministic when you do not properly manage the $$ value
yourself!
Supported location modes:
merge : generate a parser which includes the default #$ = merged(#1..#n); location tracking action for every rule,
i.e. the rule's production 'location' is the range spanning its terms.
classic : same as merge mode.
ast : ditto.
none : JISON will produce a slightly faster parser but then you are solely responsible for propagating rule action #$ location results. The default rule location is still deterministic though, as it is set to undefined: #$ = undefined;
skip : same as "none" mode, except JISON does NOT INJECT a default location action ANYWHERE, hence rule location results are not deterministic when you do not properly manage the #$ value yourself!
Notes:
when you do specify a value default mode, but DO NOT specify a location value mode, the latter is assumed to be the same as the former.
Hence:
--default-action=ast
equals:
--default-action=ast,ast
when you do not specify an explicit default mode or only a "true"/"1" value, the default is assumed: classic,merge.
when you specify "false"/"0" as an explicit default mode, none,none is assumed. This produces the fastest deterministic parser.
Default setting: [classic,merge]
Now that we are going to 're-use' the fist_column and last_column members of yylloc instead of adding a new counter member, the magic bits that do the work remain nearly the same as in Solution 1:
augmenting the lexer in its %% section:
// lexer extra code
var token_counter = 0;
lexer.post_lex = function (token) {
++token_counter;
this.yylloc.first_column = token_counter;
this.yylloc.last_column = token_counter;
return token;
};
// extra helper so multiple parse() calls will restart counting tokens:
lexer.reset_token_counter = function () {
token_counter = 0;
};
Side Note: we 'abuse' the column part for tracking the token number; meanwhile the range member will still be usable to debug the raw text input as that one will track the positions within the raw input string.
Make sure to tweak both first_column and last_column so that the default location tracking 'merging' code in the generated parser can still do its job: that way we'll get to see which range of tokens constitute a particular grammar rule/element, just like
it were text columns.
Could've done the same with first_line/last_line, but I felt it more suitable to use the column part for this as it's at the same very low granularity level as 'token index'...
We hook a pre_lex callback into the flow by directly injecting it into the lexer definition via lexer.post_lex = function (){...}.
Same as Solution 1, now all we have to do is complete this with a tiny bit of pre_parse code to ensure multiple parser.parse(input) invocations each will restart with the token counter reset to zero:
// extra helper: reset the token counter at the start of every parse() call:
parser.pre_parse = function (yy) {
yy.lexer.reset_token_counter();
};
Of course, that bit has to be added to the parser's final code block, after the %% in the grammar spec part of the jison file.
Full jison source file is available as a gist here.
How to compile and test:
# compile
jison --main so-q-58891186-3.jison
# run test code in main()
node so-q-58891186-3.js
Aftermath / Observations about the solutions provided
Observe the test verification data at the end of both those jison files provided for how the token index shows up in the parser output:
Solution 1 (stripped, partial) output:
"type": "ProgramStmt",
"a1": [
{
"type": "ExprStmt",
"a1": {
"type": "AssignmentValueExpr",
"target": {
"type": "VariableRefExpr",
"a1": "ABA0",
"loc": {
"range": [
0,
8
],
"counter": 1
}
},
"source": {
"type": "VariableRefExpr",
"a1": "X",
"loc": {
"counter": 6
}
},
"loc": {
"counter": 1
}
},
"loc": {
"counter": 1
}
}
],
"loc": {
"counter": 1
}
Note here that the counter index is not really accurate for compound elements, i.e. elements which were constructed from multiple tokens matching one or more grammar rules: only the first token index is kept.
Solution 2 fares much better in that regard:
Solution 2 (stripped, partial) output:
"type": "ExprStmt",
"a1": {
"type": "AssignmentValueExpr",
"target": {
"type": "VariableRefExpr",
"a1": "ABA0",
"loc": {
"first_column": 1,
"last_column": 4,
}
},
"source": {
"type": "VariableRefExpr",
"a1": "X",
"loc": {
"first_column": 6,
"last_column": 6,
}
},
"loc": {
"first_column": 1,
"last_column": 6,
}
},
"loc": {
"first_column": 1,
"last_column": 7,
}
}
As you can see the first_column plus last_column members nicely track the set of tokens which constitute each part.
(Note that the counter increment code implied we start counting with ONE(1), not ZERO(0)!)
Parting thought
Given the input A;B;A;D0;ASSIGN;X;SEMICOLON; the current grammar parses this like ABA0 = X; and I wonder if this is what you really intend to get: constructing the identifier ABA0 like that seems a little odd to me.
Alas, that's not relevant to your question. It's just me encountering something quite out of the ordinary here, that's all. No matter.
Cheers and hope this long blurb is helpful to more of us. :-)
Source files:
original OP file as gist
solution 1 JISON file
solution 2 JISON file
current jison-gho release example grammars, including several which demo advanced features
Is there any way of storing line numbers in the created parse tree, using ANTLR 4? I came across this article, which does it but I think it's for older ANTLR version, because
parser.setASTFactory(factory);
It does not seem to be applicable to ANTLR 4.
I am thinking of having something like
treenode.getLine()
, like we can have
treenode.getChild()
With Antlr4, you normally implement either a listener or a visitor.
Both give you a context where you find the location of the tokens.
For example (with a visitor), I want to keep the location of an assignment defined by a Uppercase identifier (UCASE_ID in my token definition).
The bit you're interested in is ...
ctx.UCASE_ID().getSymbol().getLine()
The visitor looks like ...
static class TypeAssignmentVisitor extends ASNBaseVisitor<TypeAssignment> {
#Override
public TypeAssignment visitTypeAssignment(TypeAssignmentContext ctx) {
String reference = ctx.UCASE_ID().getText();
int line = ctx.UCASE_ID().getSymbol().getLine();
int column = ctx.UCASE_ID().getSymbol().getCharPositionInLine()+1;
Type type = ctx.type().accept(new TypeVisitor());
TypeAssignment typeAssignment = new TypeAssignment();
typeAssignment.setReference(reference);
typeAssignment.setReferenceToken(new Token(ctx.UCASE_ID().getSymbol().getLine(), ctx.UCASE_ID().getSymbol().getCharPositionInLine()+1));
typeAssignment.setType(type);
return typeAssignment;
}
}
I was new to Antlr4 and found this useful to get started with listeners and visitors ...
https://github.com/JakubDziworski/AntlrListenerVisitorComparison/
I have text that is already tokenized, sentence-split, and POS-tagged.
I would like to use CoreNLP to additionally annotate lemmas (lemma), named entities (ner), contituency and dependency parse (parse), and coreferences (dcoref).
Is there a combination of commandline options and option file specifications that makes this possible from the command line?
According to this question, I can ask the parser to view whitespace as delimiting tokens, and newlines as delimiting sentences by adding this to my properties file:
tokenize.whitespace = true
ssplit.eolonly = true
This works well, so all that remains is to specify to CoreNLP that I would like to provide POS tags too.
When using the Stanford Parser standing alone, it seems to be possible to have it use existing POS tags, but copying that syntax to the invocation of CoreNLP doesn't seem to work. For example, this does not work:
java -cp *:./* -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -props my-properties-file -outputFormat xml -outputDirectory my-output-dir -sentences newline -tokenized -tagSeparator / -tokenizerFactory edu.stanford.nlp.process.WhitespaceTokenizer -tokenizerMethod newCoreLabelTokenizerFactory -file my-annotated-text.txt
While this question covers programmatic invocation, I'm invoking CoreNLP form the commandline as part of a larger system, so I'm really asking whether this is possible to achieve this with commandline options.
I don't think this is possible with command line options.
If you want you can make a custom annotator and include it in your pipeline you could go that route.
Here is some sample code:
package edu.stanford.nlp.pipeline;
import edu.stanford.nlp.util.logging.Redwood;
import edu.stanford.nlp.ling.*;
import edu.stanford.nlp.util.concurrent.MulticoreWrapper;
import edu.stanford.nlp.util.concurrent.ThreadsafeProcessor;
import java.util.*;
public class ProvidedPOSTaggerAnnotator {
public String tagSeparator;
public ProvidedPOSTaggerAnnotator(String annotatorName, Properties props) {
tagSeparator = props.getProperty(annotatorName + ".tagSeparator", "_");
}
public void annotate(Annotation annotation) {
for (CoreLabel token : annotation.get(CoreAnnotations.TokensAnnotation.class)) {
int tagSeparatorSplitLength = token.word().split(tagSeparator).length;
String posTag = token.word().split(tagSeparator)[tagSeparatorSplitLength-1];
String[] wordParts = Arrays.copyOfRange(token.word().split(tagSeparator), 0, tagSeparatorSplitLength-1);
String tokenString = String.join(tagSeparator, wordParts);
// set the word with the POS tag removed
token.set(CoreAnnotations.TextAnnotation.class, tokenString);
// set the POS
token.set(CoreAnnotations.PartOfSpeechAnnotation.class, posTag);
}
}
}
This should work if you provide your token with POS tokens separated by "_". You can change it with the forcedpos.tagSeparator property.
If you set customAnnotator.forcedpos = edu.stanford.nlp.pipeline.ProvidedPOSTaggerAnnotator
to the property file, include the above class in your CLASSPATH, and then include "forcedpos" in your list of annotators after "tokenize", you should be able to pass in your own pos tags.
I may clean this up some more and actually include it in future releases for people!
I have not had time to actually test this code out, if you try it out and find errors please let me know and I'll fix it!