Parse arbitary user input [closed] - parsing

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I have a database full of messages from a bulletin board. The board uses BB codes as formatting style. I.e.:
I'm not formatted
This is [b]bold[/b] text
Tags can also [i][b]be[/b] nested[/i]
And the [b]nesting [i]can be[/b] rather[/i] ugly
My ultimate goal is to convert these messages to some well formed XML (no discussion here ;) ). I don't want to use regular expression, which will fail at some point (in fact: it does).
First step: parse a message into some kind of internal representation (a graph, a tree, etc.). And I'm stuck at this point. The actual extraction is not that big problem, but the storage is.
How do I represent this kind of markup into some meaningful structure. My problem seems to be similar (or almost identical) to a browser building a DOM from a HTML file. So I think there are some strategies to solve it. I know the solution will not be perfect but im willing to invest a vast amount of time to do build the best possible.
Question: Do you have any tips/hint/comments? Any articles or paper you can recommend? Or a book which discusses these topic? I'm grateful for any input.

Answer 1

And the [b]nesting [i]can be[/b] rather[/i] ugly
I've written a parser very similar to what you are looking to do except that it would throw an error on your fourth example. Something to the effect of "Unexpected end tag [/b] within [i]".
I think that what you want to do is very doable but internally you will want to create a tree as if your original text was:
"And the [b]nesting [i]can be[/i][/b][i] rather[/i] ugly". (I don't think this would be necessary if you didn't need to convert it to XML later. If there were no need to convert to XML you could keep a linked list of text sections where each section is marked with its format combination)
Two possible approaches to this problem come to mind (of course there could be better possibilities). 1) Preprocess and insert the missing end and begin tags where necessary. 2) Build your parse tree and where there are overlapping tags imply the missing ones based on the current context. I think approach number (2) would be simpler and cleaner.
You could model your tree based on a composite pattern where you have an AbstractElement class, a TextElement class that extends AbstractElement, and a Tag class that extends AbstractElement and contains a list of sub-elements of type AbstractElement.
You would start by creating a root Tag instance. You would then call rootTag.parse(text). You would need a scanner that could return 3 types of tokens: text, start-tags, and end-tags. The scanner would allow you to push tokens onto it, which it would return before any normal scanned token. This would allow you to push new start tag tokens on after encountering and dealing with the unexpected end tag. You would also have to know when you are done with input. I'll use a 4th token type for that.
/* methods within class Tag */
public void parse(String text) {
MyScanner scanner = new MyScanner(text);
parse(scanner);
}
/* returns next token */
private Token parse(MyScanner scanner) {
Token firstToken = scanner.getNextToken();
return parse(scanner,firstToken);
}
private Token parse(MyScanner scanner) {
Token firstToken = scanner.getNextToken();
return parse(scanner,firstToken);
}
private Token parse(MyScanner scanner, Token token) {
while (!token.isDone() && !token.isEndTag()) {
if (token.isStartTag()) {
Tag subTag = new Tag(token.getValue());
token = scanner.getNextToken();
token = subTag.parse(scanner,token);
addElement(subTag);
}
else {
TextElement text = new TextElement(token.getValue());
addElement(text);
token = scanner.getNextToken();
}
}
if (token.isEndTag()) {
if (!token.getValue().equals(getName()) {
scanner.push(new Token(Token.START_TAG,token.getValue()));
}
else {
token = scanner.getNextToken();
}
}
return token;
}
So if you were to parse "And the [b]nesting [i]can be[/b] rather[/i] ugly", The following should get created.
rootTag.parse should be adding:
TextElement: "And the "
Tag: "b"
TextElement: "nesting "
Tag: "i"
TextElement: "can be"
(... at this point the odd [/b] is encountered ...)
(... push "i" start tag on the scanner ...)
(... here the [/b] is encountered (again) ...)
Tag: "i" (this was scanned because it had been pushed to the scanner)
TextElement: " rather"
TextElement: " ugly"
Note: Coding within a text area does not lend itself well to testing and debugging. Accept this answer as a hint or a possibility, not as your definate answer.