Binary Protocol Parsing in Erlang - parsing

Question

I am a bit struggling with extracting fields from a binary message. Raw message looks like the following:
<<1,0,97,98,99,100,0,0,0,3,0,0,0,0,0,0,0,0,0,3,32,3,0,0,88,2,0,0>>
I know the order, type and static sizes of fields, some have arbitary sizes thought, so I am trying to do something like the following:
newobj(Data) ->
io:fwrite("NewObj RAW ~p~n",[Data]),
NewObj = {obj,rest(uint16(string(uint16({[],Data},id),type),parent),unparsed)},
io:fwrite("NewObj ~p~n",[NewObj]),
NewObj.
uint16/2, string/2, and rest/2 are actually extraction functions and look like this:
uint16(ListData, Name) ->
{List, Data} = ListData,
case Data of
<<Int:2/little-unsigned-unit:8, Rest/binary>> ->
{List ++ [{Name,Int}], Rest};
<<Int:2/little-unsigned-unit:8>> ->
List ++ [{Name,Int}]
end.
string(ListData, Name) ->
{List, Data} = ListData,
Split = binary:split(Data,<<0>>),
String = lists:nth(1, Split),
if
length(Split) == 2 ->
{List ++ [{Name, String}], lists:nth(2, Split)};
true ->
List ++ [{Name, String}]
end.
rest(ListData, Name) ->
{List, Data} = ListData,
List ++ [{Name, Data}].
This works and looks like:
NewObj RAW <<1,0,97,98,99,100,0,0,0,3,0,0,0,0,0,0,0,0,0,3,32,3,0,0,88,2,0,0>>
NewObj {obj,[{id,1},
{type,<<"abcd">>},
{parent,0},
{unparsed,<<3,0,0,0,0,0,0,0,0,0,3,32,3,0,0,88,2,0,0>>}]}
The reason for this question though is that passing {List, Data} as ListData and then splitting it within the function with {List, Data} = ListData feels clumsy - so is there a better way? I think I can't use static matching because "unparsed" and "type" parts are of arbitary length, so it's not possible to define their respective sizes.
Thanks!
---------------Update-----------------
Trying to take comments below into account - code now looks like the following:
newobj(Data) ->
io:fwrite("NewObj RAW ~p~n",[Data]),
NewObj = {obj,field(
field(
field({[], Data},id,fun uint16/1),
type, fun string/1),
unparsed,fun rest/1)},
io:fwrite("NewObj ~p~n",[NewObj]).
field({List, Data}, Name, Func) ->
{Value,Size} = Func(Data),
case Data of
<<_:Size/binary-unit:8>> ->
[{Name,Value}|List];
<<_:Size/binary-unit:8, Rest/binary>> ->
{[{Name,Value}|List], Rest}
end.
uint16(Data) ->
case Data of
<<UInt16:2/little-unsigned-unit:8, _/binary>> ->
{UInt16,2};
<<UInt16:2/little-unsigned-unit:8>> ->
{UInt16,2}
end.
string(Data) ->
Split = binary:split(Data,<<0>>),
case Split of
[String, Rest] ->
{String,byte_size(String)+1};
[String] ->
{String,byte_size(String)+1}
end.
rest(Data) ->
{Data,byte_size(Data)}.

Answer 1

The code is non idiomatic and some pieces cannot compile as is :-) Here are some comments:
The newobj/1 function makes a reference to a NewObj variable that is unbound. Probably the real code is something like NewObj = {obj,rest(... ?
The code uses list append (++) multiple times. This should be avoided if possible because it performs too much memory copies. The idiomatic way is to add to the head of the list as many times as needed (that is: L2 = [NewThing | L1]) and call lists:reverse/1 at the very end. See any Erlang book or the free Learn Yourself some Erlang for the details.
In a similar vein, lists:nth/2 should be avoided and replaced by pattern matching or a different way to construct the list or parse the binary
Dogbert's suggestion about doing the pattern matching directly in the function argument is a good idiomatic approach and allows to remove some lines from the code.
As last suggestion regarding the approach to debug, consider replacing the fwrite functions with proper unit tests.
Hope this gives some hints for what to look at. Feel free to append to your question the code changes, we can proceed from there.
EDIT
It's looking better. Let's see if we can simplify. Please note that we are doing the work backwards, because we are adding tests after the production code has been written, instead of doing test-driven development.
Step 1: add test.
I also reversed the order of the list because it looks more natural.
-include_lib("eunit/include/eunit.hrl").
happy_input_test() ->
Rest = <<3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 32, 3, 0, 0, 88, 2, 0, 0>>,
Input = <<1, 0,
97, 98, 99, 100, 0,
0, 0,
Rest/binary>>,
Expected = {obj, [{id, 1}, {type, <<"abcd">>}, {parent, 0}, {unparsed, Rest}]},
?assertEqual(Expected, binparse:newobj(Input)).
We can run this, among other ways, with rebar3 eunit (see the rebar3 documentation; I suggest to start with rebar3 new lib mylib to create a skeleton).
Step 2: the absolute minimum
Your description is not enough to understand which fields are mandatory and which are optional and whether there is always something more after the obj.
In the simplest possible case, all your code can be reduced to:
newobj(Bin) ->
<<Id:16/little-unsigned, Rest/binary>> = Bin,
[Type, Rest2] = binary:split(Rest, <<0>>),
<<Parent:16/little-unsigned, Rest3/binary>> = Rest2,
{obj, [{id, Id}, {type, Type}, {parent, Parent}, {unparsed, Rest3}]}.
Quite compact :-)
I find the encoding of the string very bizarre: a binary encoding where the string is NUL-terminated (so forces to walk the binary) instead of being encoded with, say, 2 or 4 bytes to represent the length and then the string itself.
Step 3: input validation
Since we are parsing a binary, this is probably coming from the outside of our system. As such, the let it crash philosophy doesn't apply and we have to perform full input validation.
I make the assumption that all fields are mandatory except unparsed, that can be empty.
missing_unparsed_is_ok_test() ->
Input = <<1, 0,
97, 98, 99, 100, 0,
0, 0>>,
Expected = {obj, [{id, 1}, {type, <<"abcd">>}, {parent, 0}, {unparsed, <<>>}]},
?assertEqual(Expected, binparse:newobj(Input)).
The simple implementation above passes it.
Step 4: malformed parent
We add the tests and we make a API decision: the function will return an error tuple.
missing_parent_is_error_test() ->
Input = <<1, 0,
97, 98, 99, 100, 0>>,
?assertEqual({error, bad_parent}, binparse:newobj(Input)).
malformed_parent_is_error_test() ->
Input = <<1, 0,
97, 98, 99, 100, 0,
0>>,
?assertEqual({error, bad_parent}, binparse:newobj(Input)).
We change the implementation to pass the tests:
newobj(Bin) ->
<<Id:16/little-unsigned, Rest/binary>> = Bin,
[Type, Rest2] = binary:split(Rest, <<0>>),
case Rest2 of
<<Parent:16/little-unsigned, Rest3/binary>> ->
{obj, [{id, Id}, {type, Type}, {parent, Parent}, {unparsed, Rest3}]};
Rest2 ->
{error, bad_parent}
end.
Step 5: malformed type
The new tests:
missing_type_is_error_test() ->
Input = <<1, 0>>,
?assertEqual({error, bad_type}, binparse:newobj(Input)).
malformed_type_is_error_test() ->
Input = <<1, 0,
97, 98, 99, 100>>,
?assertEqual({error, bad_type}, binparse:newobj(Input)).
We could be tempted to change the implementation as follows:
newobj(Bin) ->
<<Id:16/little-unsigned, Rest/binary>> = Bin,
case binary:split(Rest, <<0>>) of
[Type, Rest2] ->
case Rest2 of
<<Parent:16/little-unsigned, Rest3/binary>> ->
{obj, [
{id, Id}, {type, Type},
{parent, Parent}, {unparsed, Rest3}
]};
Rest2 ->
{error, bad_parent}
end;
[Rest] -> {error, bad_type}
end.
Which is an unreadable mess. Just adding functions doesn't help us:
newobj(Bin) ->
<<Id:16/little-unsigned, Rest/binary>> = Bin,
case parse_type(Rest) of
{ok, {Type, Rest2}} ->
case parse_parent(Rest2) of
{ok, Parent, Rest3} ->
{obj, [
{id, Id}, {type, Type},
{parent, Parent}, {unparsed, Rest3}
]};
{error, Reason} -> {error, Reason}
end;
{error, Reason} -> {error, Reason}
end.
parse_type(Bin) ->
case binary:split(Bin, <<0>>) of
[Type, Rest] -> {ok, {Type, Rest}};
[Bin] -> {error, bad_type}
end.
parse_parent(Bin) ->
case Bin of
<<Parent:16/little-unsigned, Rest/binary>> -> {ok, Parent, Rest};
Bin -> {error, bad_parent}
end.
This is a classic problem in Erlang with nested conditionals.
Step 6: regaining sanity
Here is my approach, quite generic so applicable (I think) to many domains. The overall idea is taken from backtracking, as explained in http://rvirding.blogspot.com/2009/03/backtracking-in-erlang-part-1-control.html
We create one function per parse step and pass them, as a list, to call_while_ok/3:
newobj(Bin) ->
Parsers = [fun parse_id/1,
fun parse_type/1,
fun parse_parent/1,
fun(X) -> {ok, {unparsed, X}, <<>>} end
],
case call_while_ok(Parsers, Bin, []) of
{error, Reason} -> {error, Reason};
PropList -> {obj, PropList}
end.
Function call_while_ok/3 is somehow related to lists:foldl and lists:filter:
call_while_ok([F], Seed, Acc) ->
case F(Seed) of
{ok, Value, _NextSeed} -> lists:reverse([Value | Acc]);
{error, Reason} -> {error, Reason}
end;
call_while_ok([F | Fs], Seed, Acc) ->
case F(Seed) of
{ok, Value, NextSeed} -> call_while_ok(Fs, NextSeed, [Value | Acc]);
{error, Reason} -> {error, Reason}
end.
And here are the parsing functions. Note that their signature is always the same:
parse_id(Bin) ->
<<Id:16/little-unsigned, Rest/binary>> = Bin,
{ok, {id, Id}, Rest}.
parse_type(Bin) ->
case binary:split(Bin, <<0>>) of
[Type, Rest] -> {ok, {type, Type}, Rest};
[Bin] -> {error, bad_type}
end.
parse_parent(Bin) ->
case Bin of
<<Parent:16/little-unsigned, Rest/binary>> ->
{ok, {parent, Parent}, Rest};
Bin -> {error, bad_parent}
end.
Step 7: homework
The list [{id, 1}, {type, <<"abcd">>}, {parent, 0}, {unparsed, Rest}] is a proplist (see Erlang documentation), which predates Erlang maps.
Have a look at the documentation for maps and see if it makes sense to return a map instead.