%%
[0-9]+ {printf("%s is a number\n",yytext);}
%%
int main()
{
yylex();
}
For input affafe it results affafe, but it is not matched. How can I stop showing unmatched input in output?
(F)lex inserts a default action to handle the case where no pattern matches. The default action matches a single character and echoes it to standard output.
To avoid that, add your own fallback rule at the end of your patterns which matches a single character and does nothing:
.|\n ;
Related
my lexical analyzer in flex can not recognize numbers and ids and operators ,only keywords were recognized where is my mistake? this is my code:
%{
#include<stdio.h>
%}
Nums [0-9]
LowerCase [a-z]
UpperCase [A-Z]
Letters LowerCase|UpperCase|[_]
Id {Letters}({Letters}|{Nums})*
operators +|-|\|*
%%
"if" {printf("if keyword founded \n");}
"then" {printf("then keyword founded \n");}
"else" {printf("else keyword founded \n");}
Operators {printf(" operator founded \n");}
Id {printf(" id founded ");}
%%
int main (void)
{ yylex(); return(0);}
int yywrap(void)
{ return 1;}
The pattern Operators is equivalent to "Operators", so it only matches that single word. If you meant to expand the macro by that name, the syntax is {Operators}. (Actually, {operators} since you seem to have inconsistently spelled the macro name in all lower-case.)
If you do that, flex will complain because of the syntax error in that macro. (Syntax errors in macros aren't detected unless the macro is expanded. That's just one of the problems with using macros.)
You have different problems with your other macros. For example, Nums doesn't appear in any rule at all.
My suggestion would be to use fewer (or no) macros and more character classes. Eg.:
[[:alpha:]_][[:alnum:]_]* { /* Action for identifier. */ }
[[:digit:]]+ { /* Action for number. */ }
[-+*/] { /* Action for operator. */ }
Please read the Patterns section in the flex manual for a full description of the pattern syntax, including the named character class expressions used in the first two patterns above.
To use a named definition, it ust be enclosed in {}. So your Letters rule should be
Letters {LowerCase}|{UpperCase}|[_]
... as it is, it matches the literal inputs LowerCase and UpperCase. Similarly in your rules, you want
{Operators} ...
{Id} ...
as what you have will match the literal input strings Operators and Id
I'm trying to make a regular expression that will only work when a valid identifier name is given, using flex (the name cannot start with a number). I'm using this code :
%{
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
%}
%%
"if" { printf("IF "); }
[a-zA-Z_][a-zA-Z_0-9]* { printf("%s ", yytext); }
%%
int main() {
yylex();
}
but it is not working. how to make sure that flex accepts only a valid identifier?
When I provide the input:
if
abc
9abc
I see the following output:
IF
abc
9abc
but I expected:
IF
abc
(nothing)
Your patterns do not match all possible inputs.
In such cases, (f)lex adds a default catch-all rule, of the form
.|\n { ECHO; }
In other words, any character not recognized by your patterns will simply be printed on stdout. That will be the case with the newline characters in your input, as well as with the digit 9. After the 9 is recognized by the default rule, the remaining input will again be recognized by your identifier rule.
So you probably wanted something like this:
%option warn nodefault
%%
[[:space:]]+ ; /* Ignore whitespace */
"if" { /* TODO: Handle an "if" token */ }
[[:alpha:]_][[:alnum:]_]* { /* TODO: Handle an identifier token */ }
. { /* TODO: Handle an error */ }
Instead of printing information to stdout in an action as a debugging or learning aid, I strongly suggest you use the -T (or --trace) option when you are building your scanner. That will automatically output debugging information in a consistent and complete manner; it would have told you that the default rule was being matched, for example.
Notes:
%option nodefault tells flex not to insert a default rule. I recommend always using it, because it will keep you out of trouble. The warn option ensures that a warning is issued in this case; I think that warn is default flex behaviour but the manual suggests using it and it cannot hurt.
It's good style to use standard character class expressions. Inside a character class ([…]), [:xxx:] matches anything for which the standard library function isxxx would return true. So [[:space:]]+ matches one or more whitespace characters, including space, tab, and newline (and some others), [[:alpha:]_] matches any letter or an underscore, and [[:alnum:]_]* matches any number (including 0) of letters, digits, or underscores. See the Patterns section of the manual.
I'm taking a course in compiler construction, and my current assignment is to write the lexer for the language we're implementing. I can't figure out how to satisfy the requirement that the lexer must recognize concatenated tokens. That is, tokens not separated by whitespace. E.g.: the string 39if is supposed to be recognized as the number 39 and the keyword if. Simultaneously, the lexer must also exit(1) when it encounters invalid input.
A simplified version of the code I have:
%{
#include <stdio.h>
%}
%option main warn debug
%%
if |
then |
else printf("keyword: %s\n", yytext);
[[:digit:]]+ printf("number: %s\n", yytext);
[[:alpha:]][[:alnum:]]* printf("identifier: %s\n", yytext);
[[:space:]]+ // skip whitespace
[[:^space:]]+ { printf("ERROR: %s\n", yytext); exit(1); }
%%
When I run this (or my complete version), and pass it the input 39if, the error rule is matched and the output is ERROR: 39if, when I'd like it to be:
number: 39
keyword: if
(I.e. the same as if I entered 39 if as the input.)
Going by the manual, I have a hunch that the cause is that the error rule matches a longer possible input than the number and keyword rules, and flex will prefer it. That said, I have no idea how to resolve this situation. It seems unfeasible to write an explicit regexp that will reject all non-error input, and I don't know how else to write a "catch-all" rule for the sake of handling lexer errors.
UPDATE: I suppose I could just make the catch-all rule be . { exit(1); } but I'd like to get some nicer debug output than "I got confused on line 1".
You're quite right that you should just match a single "any" character as a fallback. The "standard" way of getting information about where in the line the parsing is at is to use the --bison-bridge option, but that can be a bit of a pain, particularly if you're not using bison. There are a bunch of other ways -- look in the manual for the ways to specify your own i/o functions, for example, -- but the all around simplest IMHO is to use a start condition:
%x LEXING_ERROR
%%
// all your rules; the following *must* be at the end
. { BEGIN(LEXING_ERROR); yyless(1); }
<LEXING_ERROR>.+ { fprintf(stderr,
"Invalid character '%c' found at line %d,"
" just before '%s'\n",
*yytext, yylineno, yytext+1);
exit(1);
}
Note: Make sure that you've ignored whitespace in your rules. The pattern .+ matches any number but at least one non-newline character, or in other words up to the end of the current line (it will force flex to read that far, which shouldn't be a problem). yyless(n) backs up the read pointer by n characters, so after the . rule matches, it will rescan that character producing (hopefully) a semi-reasonable error message. (It won't really be reasonable if your input is multibyte, or has weird control characters, so you could write more careful code. Up to you. It also might not be reasonable if the error is at the end of a line, so you might also want to write a more careful regex which gets more context, and maybe even limits the number of forward characters read. Lots of options here.)
Look up start conditions in the flex manual for more info about %x and BEGIN
I am trying to use the regular expression (?r-s:pattern) as mentioned in the Flex manual.
Following code works only when i input small letter 'a' and not the caps 'A'
%%
[(?i:a)] { printf("color"); }
\n { printf("NEWLINE\n"); return EOL;}
. { printf("Mystery character %s\n", yytext); }
%%
OUTPUT
a
colorNEWLINE
A
Mystery character A
NEWLINE
Reverse is also true i.e. if i change the line (?i:a) to (?i:A) it only considers 'A' as valid input and not 'a'.
If I remove the square brackets i.e. [] it gives error as
"ex1.lex", line 2: unrecognized rule
If I enclose the "(?i:a)" then it compiles but after executing it always goes to last rule i.e. "Mystery character..."
Please let me know how to use it properly.
I guess I am late.. :) Anyway, which flex version are you using, I have version 2.5.35 installed and correctly recognizes above pattern. Perhaps you're using old version!!!
Now regarding the enclosing with [] brackets. It works because as per [] regex rule it will try to match any of individual (, ?, i, :, a or ). Thats why a gets recognized and not A (because it is not in the list).
The way I read the manual, the rule without the square brackets should perform the case-insensitive matching you're looking for--I can't explain why you get an error at compile time. But you can achieve the same behavior in one of two ways. One, you can enumerate the upper and lower case characters in the character class:
%%
[Aa] { printf("color"); }
%%
Two, you can specify the case-insensitive scanner option, either on the command line as -i or --case-insensitive or in your .l file:
%%
%option case-insensitive
[a] {printf("color"); }
%%
I am using flex(the lexer) to do some lexical analysis.
What I need is:
If none of the rules are matched, then a value is returned to indicate such thing has happened.
This is like the default syntax in the switch control flow structure in many programming language.
Is there a way to do such kind of stuff?
EDIT 1:
Reference from the official doc
If no match is found, then the default rule is executed:
the next character in the input is considered matched and copied to the standard output.
But how can I change the default rule?
In acacia-lex it is done in the following way:
Lexer has run method:
#Override
public void run() {
Token token;
while ((token = this.findNext()).isFound()) {
System.out.println("LEXER RES = " + token.toString());
}
}
When nothing is found, there is no default rule. Lexer method run just completed its job.
To continue lexing, at the end of tokens specification is needed token "DOT" -> ".". So if no other tokens match, DOT will match and Lexer run will continue its job.
The default rule only applies if no other rule matches. So you can simply insert your own rule which matches any single character as the last rule:
.|\n { /* Your default action. */ }
It must go at the end because (F)lex will give priority to earlier rules in the file which have the same match. You need to explicitly mention \n (unless you are certain that some other rule will match it) because in (F)lex, . matches any character except a newline.
If you are using Flex, and you don't want the default rule to ever be used, it is advisable to put
%option nodefault
into your prologue. That will suppress the default rule and produce a warning if there is some input which might not match any rule. (If you ignore the warning, a runtime error will be produced for such input.)