How to read 2 bytes in flex? - flex-lexer

Question

I want to read utf-16 by 2 bytes in flex and use it.
I will use 2 byte data converted to multibyte.
I opened the file with "rb" and read it.
But I can't read the following.
4C AE C0 C9 20 00 00 B3 9C B0 ...
flex read like this.
4C AE C0 C9 20 00 00 B3 9C B0 ...
I tried to read with this code.
<string>[\x00-\xff][\x00-\xff]{
...
}
How to read 2 bytes at a time in flex?

Answer 1

I couldn't reproduce the problem using the hints in the question. It appears to work perfectly, processed with several different Flex versions (2.6.4, 2.5.39 and 2.5.4) in case you are using some antiquated version. I didn't try any non-default command-line options.
So it's impossible to say what you are doing wrong without seeing everything you are doing. Of course, we don't want to wade through your entire program; instead, we ask that you provide a minimal reproducible example (MRE). Note that a MRE is not your program, nor is it a few lines of your program. It's a complete, compilable program which shows the same problem. Writing a MRE and verifying that it shows the same issue is a good debugging technique; in many cases, the exercise will let you find the problem yourself.
That description might be a bit theoretical, so I'm including the test code I wrote. Had it produced the output you claim to receive, it would have served as an MRE, so it might serve as a starting point for your investigation:
%option noinput nounput noyywrap nodefault
%x U16STRING
%%
u["] { fputs("UTF-16LE string:", stdout);
BEGIN(U16STRING); }
.|\n ; /* Ignore everything else */
<U16STRING>{
\x22\x00 { putchar('\n'); BEGIN(INITIAL); }
[\x00-\xff]{2} { fputs(" <", stdout);
for (int i = 0; i < yyleng; ++i)
printf(" %02hhx", (unsigned char)yytext[i]);
fputs(" >", stdout);
}
[\x00-\xff] { printf(" %02x !!", yytext[0]); BEGIN(INITIAL); }
<<EOF>> { puts(" EOF"); return 0; }
}
That code identifies sequences which start with Ascii/UTF-8 u" and end with a UTF-16LE double quote; the sequences are expected to be UTF-16LE, although the code doesn't check for surrogates. All it does is dump the bytes in hex to demonstrate that the input is being correctly tokenised.
Here's a sample run, including the build commands (the Flex source is called b2.l):
$ flex --version
flex 2.6.4
$ flex -o b2.c b2.l
$ gcc -o b2 -Wall b2.c -lfl
$ printf 'u"\x4c\xae\xc0\xc9\x20\x00\x00\xb3\x9c\xb0\x22\x00' | ./b2
UTF-16LE string: < 4c ae > < c0 c9 > < 20 00 > < 00 b3 > < 9c b0 >
$