Why is the following saying variable unbound?
9> {<<A:Length/binary, Rest/binary>>, Length} = {<<1,2,3,4,5>>, 3}.
* 1: variable 'Length' is unbound
It's pretty clear that Length should be 3.
I am trying to have a function with similar pattern matching, ie.:
parse(<<Body:Length/binary, Rest/binary>>, Length) ->
But if fails with the same reason. How can I achieve the pattern matching I want?
What I am really trying to achieve is parse in incoming tcp stream packets as LTV(Length, Type, Value).
At some point after I parse the the Length and the Type, I want to ready only up to Length number of bytes as the value, as the rest will probably be for the next LTV.
So my parse_value function is like this:
parse_value(Value0, Left, Callback = {Module, Function},
{length, Length, type, Type, value, Value1}) when byte_size(Value0) >= Left ->
<<Value2:Left/binary, Rest/binary>> = Value0,
Module:Function({length, Length, type, Type, value, lists:reverse([Value2 | Value1])}),
if
Rest =:= <<>> ->
{?MODULE, parse, {}};
true ->
parse(Rest, Callback, {})
end;
parse_value(Value0, Left, _, {length, Length, type, Type, value, Value1}) ->
{?MODULE, parse_value, Left - byte_size(Value0), {length, Length, type, Type, value, [Value0 | Value1]}}.
If I could do the pattern matching, I could break it up to something more pleasant to the eye.
The rules for pattern matching are that if a variable X occurs in two subpatterns, as in {X, X}, or {X, [X]}, or similar, then they have to have the same value in both positions, but the matching of each subpattern is still done in the same input environment - bindings from one side do not carry over to the other. The equality check is conceptually done afterwards, as if you had matched on {X, X2} and added a guard X =:= X2. This means that your Length field in the tuple cannot be used as input to the binary pattern, not even if you make it the leftmost element.
However, within a binary pattern, variables bound in a field can be used in other fields following it, left-to-right. Therefore, the following works (using a leading 32-bit size field in the binary):
1> <<Length:32, A:Length/binary, Rest/binary>> = <<0,0,0,3,1,2,3,4,5>>.
<<0,0,0,3,1,2,3,4,5>>
2> A.
<<1,2,3>>
3> Rest.
<<4,5>>
I've run into this before. There is some weirdness between what is happening inside binary syntax and what happens during unification (matching). I suspect that it is just that binary syntax and matching occur at different times in the VM somewhere (we don't know which Length is failing to get assigned -- maybe binary matching is always first in evaluation, so Length is still meaningless). I was once going to dig in and find out, but then I realized that I never really needed to solve this problem -- which might be why it was never "solved".
Fortunately, this won't stop you with whatever you are doing.
Unfortunately, we can't really help further unless you explain the context in which you think this kind of a match is a good idea (you are having an X-Y problem).
In binary parsing you can always force the situation to be one of the following:
Have a fixed-sized header at the beginning of the binary message that tells you the next size element you need (and from there that can continue as a chain of associations endlessly)
Inspect the binary once on entry to determine the size you are looking for, pull that one value, and then begin the real parsing task
Have a set of fields, all of predetermined sizes that conform to some a binary schema standard
Convert the binary to a list and iterate through it with any arbitrary amount of look-ahead and backtracking you might need
Quick Solution
Without knowing anything else about your general problem, a typical solution would look like:
parse(Length, Bin) ->
<<Body:Length/binary, Rest/binary>> = Bin,
ok = do_something(Body),
do_other_stuff(Rest).
But I smell something funky here.
Having things like this in your code is almost always a sign that a more fundamental aspect of the code structure is not in agreement with the data that you are handling.
But deadlines.
Erlang is all about practical code that satisfies your goals in the real world. With that in mind, I suggest that you do something like the above for now, and then return to this problem domain and rethink it. Then refactor it. This will gain you three benefits:
Something will work right away.
You will later learn something fundamental about parsing in general.
Your code will almost certainly run faster if it fits your data better.
Example
Here is an example in the shell:
1> Parse =
1> fun
1> (Length, Bin) when Length =< byte_size(Bin) ->
1> <<Body:Length/binary, Rest/binary>> = Bin,
1> ok = io:format("Chopped off ~p bytes: ~p~n", [Length, Body]),
1> Rest;
1> (Length, Bin) ->
1> ok = io:format("Binary shorter than ~p~n", [Length]),
1> Bin
1> end.
#Fun<erl_eval.12.87737649>
2> Parse(3, <<1,2,3,4,5>>).
Chopped off 3 bytes: <<1,2,3>>
<<4,5>>
3> Parse(8, <<1,2,3,4,5>>).
Binary shorter than 8
<<1,2,3,4,5>>
Note that this version is a little safer, in that we avoid a crash in the case that Length is longer than the binary. This is yet another good reason why maybe we can't do that match in the function head.
Try with below code:
{<<A:Length/binary, Rest/binary>>, _} = {_, Length} = {<<1,2,3,4,5>>, 3}.
This question is mentioned a bit in EEP-52:
Any variables used in the expression must have been previously bound, or become bound in the same binary pattern as the expression. That is, the following example is illegal:
illegal_example2(N, <<X:N,T/binary>>) ->
{X,T}.
And explained a bit more in the following e-mail: http://erlang.org/pipermail/eeps/2020-January/000636.html
Illegal. With one exception, matching is not done in a left-to-right
order, but all variables in the pattern will be bound at the same
time. That means that the variables must be bound before the match
starts. For maps, that means that the variables referenced in key
expressions must be bound before the case (or receive) that matches
the map. In a function head, all map keys must be literals.
The exception to this general rule is that within a binary pattern,
the segments are matched from left to right, and a variable bound in a
previous segment can be used in the size expression for a segment
later in the binary pattern.
Also one of the members of OTP team mentioned that they made a prototype that can do that, but it was never finished http://erlang.org/pipermail/erlang-questions/2020-May/099538.html
We actually tried to make your example legal. The transformation of
the code that we did was not to rewrite to guards, but to match
arguments or parts of argument in the right order so that variables
that input variables would be bound before being used. (We would do a
topological sort to find the correct order.) For your example, the
transformation would look similar to this:
legal_example(Key, Map) ->
case Map of
#{Key := Value} -> Value;
_ -> error(function_clause, [Key, Map])
end.
In the prototype implementation, the compiler could compile the
following example:
convoluted(Ref,
#{ node(Ref) := NodeId, Loop := universal_answer},
[{NodeId, Size} | T],
<<Int:(Size*8+length(T)),Loop>>) when is_reference(Ref) ->
Int.
Things started to fall apart when variables are repeated. Repeated
variables in patterns already have a meaning in Erlang (they should be
the same), so it become tricky to understand to distinguish between
variables being bound or variables being used a binary size or map
key. Here is an example that the prototype couldn't handle:
foo(#{K := K}, K) -> ok.
A human can see that it should be transformed similar to this:
foo(Map, K) -> case Map of
{K := V} when K =:= V -> ok end.
Here are few other examples that should work but the prototype would
refuse to compile (often emitting an incomprehensible error message):
bin2(<<Sz:8,X:Sz>>, <<Y:Sz>>) -> {X,Y}.
repeated_vars(#{K := #{K := K}}, K) -> K.
match_map_bs(#{K1 := {bin,<<Int:Sz>>}, K2 := <<Sz:8>>}, {K1,K2}) ->
Int.
Another problem was when example was correctly rejected, the error
message would be confusing.
Because much more work would clearly be needed, we have shelved the
idea for now. Personally, I am not sure that the idea is sound in the
first place. But I am sure of one thing: the implementation would be
very complicated.
UPD: latest news from 2020-05-14
Is it a kind of a small bug or made intentionally, that when I assign the same value to a variable more than once, it doesn't throw an error, but, like, assigns it again?
Here is an example:
X = 1,
X = 100 - 99,
X = 1,
X = list_to_integer("1"),
X = X.
Shouldn't it throw an error? Throwing an error could mean that some part of the code is trying to reassign the variable, and it can be just a luck that it's the same as it was before.
It's not actually an assignment, but a match operation (the equal sign is actually the match operator). See http://erlang.org/doc/reference_manual/patterns.html
What actually happens is that you are trying to "pattern match" what's on the left side of the operator (=) against what's on the right side.
If there's a variable on the left side and it's not bounded yet, it will be bounded to the value you have on the right side.
On the other hand, if the variable is already bound and the value is the same as what's on the right side of the operator, nothing happens, the matching just succeeds.
If the values are different, there's a pattern matching error.
Note that the special variable "_" will always succeed when pattern matching any value.
Pattern matching in Erlang is really helpful, because you can fail fast when something's wrong. For example in this code:
{ok, Value} = some_function()
If some_function/0 returns something like {error, _} your code will just crash and you don't risk to continue execution with invalid values.
If some_function returns the expected value, the variable Value will be now be bounded to that result, it's like having an assignment and an assertion in one line of code (if you want to look at it that way).
Makes sense?
I have this code:
-module(info).
-export([map_functions/0]).
-author("me").
map_functions() ->
{Mod,_} = code:all_loaded(),
map_functions(Mod,#{});
map_functions([H|Tail],A) ->
B = H:mod_info(exports),
map_functions(Tail,A#{H => B});
map_functions([],A) -> A.
However whenever I compile it I get a head mismatch on line 10 which is the
map_funtions([H|Tail],A) ->
I'm sure this is a very basic error but I just cannot get my head around why this does not run. It is a correct pattern match syntax [H|Tail] and the three functions with the same name but different arities are separated by commas.
Your function definition should be
map_functions() ->
{Mod,_} = code:all_loaded(),
map_functions(Mod, #{}).
map_functions([], A) -> A;
map_functions([H|Tail], A) ->
B = H:mod_info(exports),
map_functions(Tail, A#{H => B}).
The name map_functions is the same, but the arity is not. In the Erlang world that means these are two entirely different functions: map_functions/0 and map_functions/2.
Also, note that I put the "base case" first in map_functions/2 (and made the first clause's return value stick out -- breaking that to two lines is more common, but whatever). This is for three reasons: clarity, getting in the habit of writing the base case first (so you don't accidentally write infinite loops), and very often it is necessary to do this so you don't accidentally mask your base case by matching every parameter in a higher-precedence clause.
Some extended discussion on this topic is here (addressing Elixir and Erlang): Specify arity using only or except when importing function on Elixir
Function with same name but different arity are different, they are separated by dots.
code:all_loaded returns a list, so the first function should be written:
map_functions() ->
Mods = code:all_loaded(),
map_functions(Mods, #{}).
The resulting list Mods is a list of tuples of the form {ModName,BeamLocation} so the second function should be written:
map_functions([], A) -> A;
map_functions([{ModName,_}|Tail], A) ->
B = ModName:module_info(exports),
map_functions(Tail, A#{ModName => B}).
Note that you should dig into erlang libraries and try to use more idiomatic forms of code, the whole function, using list comprehension, can be written:
map_functions() ->
maps:from_list([{X,X:module_info(exports)} || {X,_} <- code:all_loaded()]).
I'd like to tidy my Eralng code, I found there're lots of issue following:
A = {Tid, _Tv0, _Tv1, Tv2, Tv3}
Is there any way to clean the code like to be: A = {Tid, SomewayReplace(4)} ???
Update1:
like #Pascal example, Is there any way to simple the code A = {T, _, _, _, _, _} like to be A = {T, SomewayReplace(4)} to replace that 4 symbol _ ???
update2
in real project, if some record include many element, I found it force me to repeat writing the symbol _, so I wonder if there is any way to simple it???
Writting A = Something means that you try to match A with Something or if A is unbound, assign Something to A. In anycase, Something must be defined.
You can find some shortcut in writting. For example, if you want to assign the result of a funtion to A, verify that the result is a tuple of 5 elements and assign the first element to T, the you can write:
A = {T,_,_,_,_} = f(Param).
The meaning of _T is exactly the same as any variable. It just says to th compiler to not issue a warning if this variable is not used in the code. It is frequent in pattern matching when you want to ignore the value of a variable but still keep trace of its meaning.
[edit]
It is not possible to write {T, SomewayReplace(4)}, but you may use records. A record is a tagged tuple (first element is the atom that identify this record. It is not shorter than placeholder for small tuples, but it is clearer, you don't need to remember the location of the information in your tuple, and it is easier to modify your code when you need to add a new element in a tuple. The syntax will be
-record(mytuple,{field1,...,fieldx,...}.
...
A = #mytuple{fieldx = T} = f(Param).
waerning: Records are managed by the compiler, so everything must be known at build time (#mytuple{Fieldx = T} is illegal, Fieldx cannot be a variable).
Why not use a record? Then you only match the fields you want to extract. As a by-effect, you make the code easier to debug, since you are forced to name the tuple by having a atom first.
How can get the type of a list. I want to execute the code if the list is proplist.
Let us say L = [a,1,b,2,c,3, ...]. Is the list L, I'm converting it to proplist like
L = [{a,1}, {b,2}, {c,3}, ...].
How can I determine whether the list is a proplist? erlang:is_list/1 is not useful for me.
You can use something like:
is_proplist([]) -> true;
is_proplist([{K,_}|L]) when is_atom(K) -> is_proplist(L);
is_proplist(_) -> false.
but necessary to consider that this function cannot be used in guards.
You'd need to check whether every element of the list is a tuple of two elements. That can be done with lists:all/2:
is_proplist(List) ->
is_list(List) andalso
lists:all(fun({_, _}) -> true;
(_) -> false
end,
List).
This depends on which definition of "proplist" you use, of course. The above is what is usually meant by "proplist", but the documentation for the proplists module says:
Property lists are ordinary lists containing entries in the form of either tuples, whose first elements are keys used for lookup and insertion, or atoms, which work as shorthand for tuples {Atom, true}.