JavaCC Token is not matched - token

Question

I'm trying to write a parser for simple language and I got to the point where I don't know how to handle this problem. Here is my.jj file
options
{
STATIC = false;
LOOKAHEAD=2;
//DEBUG_LOOKAHEAD = true;
DEBUG_TOKEN_MANAGER=true;
FORCE_LA_CHECK = true;
DEBUG_PARSER = true;
JDK_VERSION = "1.7";
}
PARSER_BEGIN(Parser)
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
public class Parser{
private static BufferedWriter bufferFileWriter;
private static FileWriter fWriter;
public static void main(String args []) throws ParseException, IOException
{
Parser parser = new Parser(System.in);
fWriter = new FileWriter("result", true);
bufferFileWriter = new BufferedWriter(fWriter);
parser.program();
// TO DO
}
}
PARSER_END(Parser)
SKIP :
{
" "
| "\r"
| "\t"
| "\n"
}
TOKEN : /* OPERATORS */
{
< PLUS : "+" >
| < MINUS : "-" >
| < MULTIPLY : "*" >
| < DIVIDE : "/" >
| < MODULO : "%" >
| < ASSIG : ":=" >
| < EQUAL : "==" >
| < DIFF : "!=" >
| < SMALLER : "<" >
| < GRATER : ">" >
| < S_OR_EQU: "<=" >
| < G_OR_EQU: "=>" >
}
TOKEN : /*KEY WORDS FROM LANGUAGE */
{
< VAR: "VAR">
| < BEGIN : "BEGIN" >
| < END : "END" >
| < IF : "IF" >
| < ELSE : "ELSE" >
| < THEN : "THEN" >
| < WHILE: "WHILE" >
| < DO : "DO" >
| < READ : "READ" >
| < WRITE : "WRITE" >
| < SEMICOL : ";" >
}
TOKEN :
{
< VALUE : < ID > | < NUMBER > >
| < NUMBER : (< DIGIT >)+ >
| < #DIGIT : [ "0"-"9" ] >
| < ID : (["a"-"z"])+ >
}
void program():
{}
{
varDeclarations()< BEGIN > commands() < END >
}
void varDeclarations():
{}
{
< VAR >
{
System.out.println("past VAR token");
}
(< ID >
)+
}
void commands():
{}
{
(LOOKAHEAD(3)
command())+
}
void command():
{
Token t;
}
{
assign()
|< IF >condition()< THEN >commands()< ELSE >commands()< END >
|< WHILE >condition()< DO >commands()< END >
|< READ >
t=< ID >
{
try
{
fWriter.append("LOAD "+t.image);
System.out.println("LOAD "+t.image);
}
catch(IOException e)
{
};
}
< SEMICOL >
|< WRITE >
t = < VALUE >< SEMICOL >
}
void assign():
{
Token t;
}
{
t=< ID >
{
}
< ASSIG >expression(t)< SEMICOL >
}
void condition():
{}
{
< VALUE > condOperator() < VALUE >
}
void condOperator():
{}
{
< EQUAL > | < DIFF > | < SMALLER > | < S_OR_EQU > | < GRATER > | < G_OR_EQU >
}
Token operator():
{
Token tok;
}
{
tok=< PLUS >
{
System.out.println(tok.image);
return tok;
}
|tok=< MINUS >
{
System.out.println(tok.image);
return tok;
}
|tok=< MULTIPLY >
{
System.out.println(tok.image);
return tok;
}
|tok=< DIVIDE >
{
System.out.println(tok.image);
return tok;
}
|tok=< MODULO >
{
System.out.println(tok.image);
return tok;
}
}
void expression(Token writeTo):
{
Symbol s;
Token t1, t2, t3;
}
{
t1 = < VALUE >
t2 = operator()
t3 = < VALUE >
< SEMICOL >
{
if(t2.image.equals("+"))
{
try
{
fWriter.append("ADD "+t1.image+" "+t2.image);
System.out.println("ADD "+t1.image+" "+t2.image);
}catch(IOException e)
{
}
}
}
}
Writing to file is not important at this moment.
And this is the text i want to parse:
VAR
a b
BEGIN
READ a ;
READ b ;
WHILE a != b DO
IF a < b THEN (* a <-> b *)
a := a + b ;
b := a - b ;
a := a - b ;
ELSE
END
a := a - b ;
END
WRITE a ;
END
and this is output I get from the debugger:
mother-ship $ java Parser test
Call: program
Call: varDeclarations
As you can see parser enters varDeclaration method but why can't he match token with word VAR?
I'll be grateful for any help.
#Theodore I did as you suggest but it didn't work. Maybe I'm compiling and executing the program the wrong way?
This is copy of my console:
$javacc Parser.jj
Java Compiler Compiler Version 5.0 (Parser Generator)
(type "javacc" with no arguments for help)
Reading from file Parser.jj . . .
File "TokenMgrError.java" is being rebuilt.
File "ParseException.java" is being rebuilt.
File "Token.java" is being rebuilt.
File "SimpleCharStream.java" is being rebuilt.
Parser generated successfully.
$ javac *.java
$ java Parser VAR a
Call: program
Call: varDeclarations

Answer 1

I had no problem getting your parser to recognize the "VAR" keyword. The problem is that the "a" is tokenized as a 'VALUE' token, while the parser expects an 'ID` token after the "VAR" keyword. (See input and output below.)
The rule for VALUE' has precedence over the rule forID` by virtue of being first. (See question 3.3 in the FAQ.)
What you probably should do is replace the rule you have now for VALUE with the following rule.
void Value() : {} { <ID> | <NUMBER> }
Input:
VAR
a
Output:
Call: program
Call: varDeclarations
Current character : V (86) at line 1 column 1
Possible string literal matches : { "VAR" }
Current character : A (65) at line 1 column 2
Possible string literal matches : { "VAR" }
Current character : R (82) at line 1 column 3
No more string literal token matches are possible.
Currently matched the first 3 characters as a "VAR" token.
****** FOUND A "VAR" MATCH (VAR) ******
Consumed token: <"VAR" at line 1 column 1>
past VAR token
Skipping character : \n (10)
Current character : a (97) at line 2 column 1
No string literal matches possible.
Starting NFA to match one of : { <VALUE> }
Current character : a (97) at line 2 column 1
Currently matched the first 1 characters as a <VALUE> token.
Possible kinds of longer matches : { <VALUE>, <ID> }
Current character : \n (10) at line 2 column 2
Currently matched the first 1 characters as a <VALUE> token.
Putting back 1 characters into the input stream.
****** FOUND A <VALUE> MATCH (a) ******
Return: varDeclarations
Return: program
Exception in thread "main" tokenNotMatched.ParseException: Encountered " <VALUE> "a "" at line 2, column 1.
Was expecting:
<ID> ...