What am I doing wrong with erl_parse:parse_form? - erlang

Question

I wrote a Hello-World module and compiled it successfully. And then I tried to learn the things under the hood by using erl_scan and erl_parse.
-module(useless).
-export([hello/0]).
hello() -> io:format("hello world\n").
and I type in the erl shell
{ok, S} = file:read_file("useless.erl").
and
{ok, T, _} = erl_scan:string(binary_to_list(S)).
It works fine apparently. But when I try erl_parse:parse_form(T).
It gives {error,{2,erl_parse,["syntax error before: ","'-'"]}}
What am I doing wrong?
Edit:
the module compile is helpful.
Also this is cool.

Answer 1

The function erl_parse:parse_form/1 works on one form only. So you must split the result of erl_scan:string/1 into individual forms first.
you can use erl_scan:tokens to achieve this (this code wors, but I am not sure I use the function in the right way):
-module(useless).
-export([eval/1]).
eval(File) ->
{ok, B} = file:read_file(File),
Forms = scan(erl_scan:tokens([],binary_to_list(B),1),[]),
F = fun(X) -> {ok,Y} = erl_parse:parse_form(X), Y end,
[F(X) || X <- Forms].
scan({done,{ok,T,N},S},Res) ->
scan(erl_scan:tokens([],S,N),[T|Res]);
scan(_,Res) ->
lists:reverse(Res).

Answer 2

Robert suggestion is to use the re-entrant feature of erl_scan:tokens/3.
The docs on this function are not explicit and it took me some time to understand that the initial string needed to be closed by eof. (if not the tokens function runs an endless loop).
Here is the code I finally came up with, I hope that will help others.
eval(File)->
{ok, Data} = file:read_file(File),
String=erlang:binary_to_list(Data),
scan(String).
scan(String) when is_list(String)->
scan(String++eof,[]). %% appended eof
scan({done, Result, LeftOverChars},Acc)->
scan_done(Result,LeftOverChars,Acc);
scan({more, Continuation},Acc)->
scan(erl_scan:tokens(Continuation,[],1),Acc);
scan(String,Acc) when is_list(String)->
scan(erl_scan:tokens([],String,1),Acc).
scan_done({error,ErrorMsg,_Location},_LeftOverChars,_Acc)->
ErrorMsg;
scan_done({eof,Location},LeftOverChars,Acc)->
Acc;
scan_done({ok,Tokens,Location},LeftOverChars,Acc)->
case erl_parse:parse_form(Tokens) of
{ok,R}->scan(LeftOverChars,Acc++[R]);
{error,R}->scan(LeftOverChars,R)
end.