Location in syntax trees - parsing

Question

When writing a parser, I want to remember the location of lexemes found, so that I can report useful error messages to the programmer, as in “if-less else on line 23” or ”unexpected character on line 45, character 6” or “variable not defined” or something similar. But once I have built the syntax tree, I will transform it in several ways, optimizing or expanding some kind of macros. The transformations produce or rearrange lexemes which do not have a meaningful location.
Therefore it seems that the type representing the syntax tree should come in two flavor, a flavor with locations decorating lexemes and a flavor without lexemes. Ideally we would like to work with a purely abstract syntax tree, as defined in the OCaml book:
# type unr_op = UMINUS | NOT ;;
# type bin_op = PLUS | MINUS | MULT | DIV | MOD
| EQUAL | LESS | LESSEQ | GREAT | GREATEQ | DIFF
| AND | OR ;;
# type expression =
ExpInt of int
| ExpVar of string
| ExpStr of string
| ExpUnr of unr_op * expression
| ExpBin of expression * bin_op * expression ;;
# type command =
Rem of string
| Goto of int
| Print of expression
| Input of string
| If of expression * int
| Let of string * expression ;;
# type line = { num : int ; cmd : command } ;;
# type program = line list ;;
We should be allowed to totally forget about locations when working on that tree and have special functions to map an expression back to its location (for instance), that we could use in case of emergency.
What is the best way to define such a type in OCaml or to handle lexeme positions?

Answer 1

The best way is to work always with AST nodes fully annotated with the locations. For example:
type expression = {
expr_desc : expr_desc;
expr_loc : Lexing.position * Lexing.position; (* start and end *)
}
and expr_desc =
ExpInt of int
| ExpVar of string
| ExpStr of string
| ExpUnr of unr_op * expression
| ExpBin of expression * bin_op * expression
Your idea, keeping the AST free of locations and writing a function to retrieve the missing locations is not a good idea, I believe. Such a function should require searching by pointer equivalence of AST nodes or something similar, which does not really scale.
I strongly recommend to look though OCaml compiler's parser.mly which is a full scale example of AST with locations.