How to parse a list of tokens with FParsec - f#

Question

I am trying to parse a list of tokens with FParsec, where each token is either a block of text or a tag - for example:
This is a {type of test} test, and it {succeeds or fails}
Here is the parser:
type Parser<'t> = Parser<'t, unit>
type Token =
| Text of string
| Tag of string
let escape fromString toString : Parser<_> =
pstring fromString |>> (fun c -> toString)
let content : Parser<_> =
let contentNormal = many1Satisfy (fun c -> c <> '{' && c <> '}')
let openBraceEscaped = escape "{{" "{"
let closeBraceEscaped = escape "}}" "}"
let contentEscaped = openBraceEscaped <|> closeBraceEscaped
stringsSepBy contentNormal contentEscaped
let ident : Parser<_> =
let isIdentifierFirstChar c = isLetter c || c = '_'
let isIdentifierChar c = isLetter c || isDigit c || c = '_'
spaces >>. many1Satisfy2L isIdentifierFirstChar isIdentifierChar "identifier" .>> spaces
let text = content |>> Text
let tag =
ident |> between (skipString "{") (skipString "}")
|>> Tag
let token = text <|> tag
let tokens = many token .>>. eof
the following tests work:
> run token "abc def" ;;
val it : ParserResult<Token,unit> = Success: Text "abc def"
> run token "{abc def}" ;;
val it : ParserResult<Token,unit> = Success: Tag "abc def"
but trying to run tokens results in an exception:
> run tokens "{abc} def" ;;
System.InvalidOperationException: (Ln: 1, Col: 10): The combinator 'many' was
applied to a parser that succeeds without consuming input and without
changing the parser state in any other way. (If no exception had been raised,
the combinator likely would have entered an infinite loop.)
I've gone over this stackoverflow question but nothing I've tried works. I even added the following, but I get the same exception:
let tokenFwd, tokenRef = createParserForwardedToRef<Token, unit>()
do tokenRef := choice [tag; text]
let readEndOfInput : Parser<unit, unit> = spaces >>. eof
let readExprs = many tokenFwd
let readExprsTillEnd = readExprs .>> readEndOfInput
run readExprsTillEnd "{abc} def" // System.InvalidOperationException ... The combinator 'many' was applied ...
I believe the problem is stringsSepBy in content, but I can't figure out any other way to get a string with the escaped items
Any help would be much appreciated - I have been going through this for a couple days now and can't figure it out.

Answer 1

stringsSepBy accepts zero strings, causing token to accept an empty string, causing many to complain.
I changed it to the following to verify that that was the line you need to work on.
many1 (contentNormal <|> contentEscaped) |>> fun l -> String.concat "" l
Also I got away from stringsSepBy contentNormal contentEscaped, because that says you need to match contentNormals with contentEscapeds in between them. So a{{b}}c is ok, but {{b}}, {{b}}c and a{{b}} will fail.

Answer 2

notEmpty can be used to consume input. If you're not consuming any input but letting the parser succeed then the "current position" of the parser is not moved forward, so when a statement doing that is inside a many it would go into an infinite loop without that exception. stringsSepBy is succeeding and parsing zero elements, you could use notEmpty to fail it if it gets zero elements:
stringsSepBy contentNormal contentEscaped |> notEmpty
Also, I tried to get your full example to parse, the tags can include spaces so you need to allow ident to include spaces to match that:
let isIdentifierChar c = isLetter c || isDigit c || c = '_' || c = ' '
Another little adjustment would be to only return a Token list rather than Token list * unit tuple (unit is the result of eof):
let tokens = many token .>> eof