I want to create a List of tuples in json format with jsx:encode, which is returning error:
io:format("Mylist After reverse ==========: ~n~p~n",[Mylist]),
FinalReading = lists:map(
fun(X) ->
jsx:encode([X])
end, Mylist),
Mylist is printed as follows:
[{{<<"name">>,<<"Batman">>},{<<"steps">>,1033},{<<"distance">>,830}},
{{<<"name">>,<<"Superman">>},{<<"steps">>,641},{<<"distance">>,520}}]
I'm getting this error:
09:49:24.048 [error] ** Boss Service Handler myproj_priv_mylib_websocket terminating in handle_incoming/4
for the reason error:badarg
ServiceUrl: "/websocket/mylib"
WebSocketId: <0.336.0>
SessionId : <<"a4f60118091309990af7c89c9a1acb49ee3bb08d">>
Message : <<"admin#mydomain.com:z3CMLU9jDTYdiAacNiJrMIkdp4lTf6sb">>
State : []
** Stacktrace: [{jsx_parser,value,4,[{file,"src/jsx_parser.erl"},{line,125}]},{lists,map,2,[{file,"lists.erl"},{line,1237}]},{myproj_priv_mylib_websocket,handle_incoming,5,[{file,"..."},{line,130}]},{boss_service_worker,handle_cast,2,[{file,"src/boss/boss_service_worker.erl"},{line,173}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,599}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]
It would be great if someone points out the correct code.
Thanking all in advance.
You'll have to convert the top level tuples to lists using erlang:tuple_to_list/1 and lists:map/2 so that jsx can correctly encode them to a JSON Array.
1> List = [{{<<"name">>,<<"Batman">>},{<<"steps">>,1033},{<<"distance">>,830}},
1> {{<<"name">>,<<"Superman">>},{<<"steps">>,641},{<<"distance">>,520}}].
[{{<<"name">>,<<"Batman">>},
{<<"steps">>,1033},
{<<"distance">>,830}},
{{<<"name">>,<<"Superman">>},
{<<"steps">>,641},
{<<"distance">>,520}}]
2> List2 = lists:map(fun erlang:tuple_to_list/1, List).
[[{<<"name">>,<<"Batman">>},
{<<"steps">>,1033},
{<<"distance">>,830}],
[{<<"name">>,<<"Superman">>},
{<<"steps">>,641},
{<<"distance">>,520}]]
3> io:format("~s~n", [jsx:encode(List2)]).
[{"name":"Batman","steps":1033,"distance":830},{"name":"Superman","steps":641,"distance":520}]
I'm very much a newbie to Erlang, so not sure if I've the right to judge, but my feeling is that it would be good to hide the need to convert the tuple to list in jsx:encode.
If
[{a,<<"a">>}, {b,<<"b">>}].
becomes
{
"a" : "a",
"b" : "b"
}
I believe the same should work as well for this syntax
{{a,<<"a">>}, {b,<<"b">>}}.
This last syntax brings more information (it reflects an ordering which I can count on, within the code, for pattern matching).
I usually try to convert my JSON into a tuple for which I know the ordering, so that I can play with pattern matching instead of using keyfind and friends. But because the above syntax does not work, I have to wrap jsx:encode(tuple_to_list()) to automatically manage "PropTuples".
Am I completely misunderstanding it? Is there a more decent way to represent/treat/manipulate JSON in Erlang?
Related
Why is the following saying variable unbound?
9> {<<A:Length/binary, Rest/binary>>, Length} = {<<1,2,3,4,5>>, 3}.
* 1: variable 'Length' is unbound
It's pretty clear that Length should be 3.
I am trying to have a function with similar pattern matching, ie.:
parse(<<Body:Length/binary, Rest/binary>>, Length) ->
But if fails with the same reason. How can I achieve the pattern matching I want?
What I am really trying to achieve is parse in incoming tcp stream packets as LTV(Length, Type, Value).
At some point after I parse the the Length and the Type, I want to ready only up to Length number of bytes as the value, as the rest will probably be for the next LTV.
So my parse_value function is like this:
parse_value(Value0, Left, Callback = {Module, Function},
{length, Length, type, Type, value, Value1}) when byte_size(Value0) >= Left ->
<<Value2:Left/binary, Rest/binary>> = Value0,
Module:Function({length, Length, type, Type, value, lists:reverse([Value2 | Value1])}),
if
Rest =:= <<>> ->
{?MODULE, parse, {}};
true ->
parse(Rest, Callback, {})
end;
parse_value(Value0, Left, _, {length, Length, type, Type, value, Value1}) ->
{?MODULE, parse_value, Left - byte_size(Value0), {length, Length, type, Type, value, [Value0 | Value1]}}.
If I could do the pattern matching, I could break it up to something more pleasant to the eye.
The rules for pattern matching are that if a variable X occurs in two subpatterns, as in {X, X}, or {X, [X]}, or similar, then they have to have the same value in both positions, but the matching of each subpattern is still done in the same input environment - bindings from one side do not carry over to the other. The equality check is conceptually done afterwards, as if you had matched on {X, X2} and added a guard X =:= X2. This means that your Length field in the tuple cannot be used as input to the binary pattern, not even if you make it the leftmost element.
However, within a binary pattern, variables bound in a field can be used in other fields following it, left-to-right. Therefore, the following works (using a leading 32-bit size field in the binary):
1> <<Length:32, A:Length/binary, Rest/binary>> = <<0,0,0,3,1,2,3,4,5>>.
<<0,0,0,3,1,2,3,4,5>>
2> A.
<<1,2,3>>
3> Rest.
<<4,5>>
I've run into this before. There is some weirdness between what is happening inside binary syntax and what happens during unification (matching). I suspect that it is just that binary syntax and matching occur at different times in the VM somewhere (we don't know which Length is failing to get assigned -- maybe binary matching is always first in evaluation, so Length is still meaningless). I was once going to dig in and find out, but then I realized that I never really needed to solve this problem -- which might be why it was never "solved".
Fortunately, this won't stop you with whatever you are doing.
Unfortunately, we can't really help further unless you explain the context in which you think this kind of a match is a good idea (you are having an X-Y problem).
In binary parsing you can always force the situation to be one of the following:
Have a fixed-sized header at the beginning of the binary message that tells you the next size element you need (and from there that can continue as a chain of associations endlessly)
Inspect the binary once on entry to determine the size you are looking for, pull that one value, and then begin the real parsing task
Have a set of fields, all of predetermined sizes that conform to some a binary schema standard
Convert the binary to a list and iterate through it with any arbitrary amount of look-ahead and backtracking you might need
Quick Solution
Without knowing anything else about your general problem, a typical solution would look like:
parse(Length, Bin) ->
<<Body:Length/binary, Rest/binary>> = Bin,
ok = do_something(Body),
do_other_stuff(Rest).
But I smell something funky here.
Having things like this in your code is almost always a sign that a more fundamental aspect of the code structure is not in agreement with the data that you are handling.
But deadlines.
Erlang is all about practical code that satisfies your goals in the real world. With that in mind, I suggest that you do something like the above for now, and then return to this problem domain and rethink it. Then refactor it. This will gain you three benefits:
Something will work right away.
You will later learn something fundamental about parsing in general.
Your code will almost certainly run faster if it fits your data better.
Example
Here is an example in the shell:
1> Parse =
1> fun
1> (Length, Bin) when Length =< byte_size(Bin) ->
1> <<Body:Length/binary, Rest/binary>> = Bin,
1> ok = io:format("Chopped off ~p bytes: ~p~n", [Length, Body]),
1> Rest;
1> (Length, Bin) ->
1> ok = io:format("Binary shorter than ~p~n", [Length]),
1> Bin
1> end.
#Fun<erl_eval.12.87737649>
2> Parse(3, <<1,2,3,4,5>>).
Chopped off 3 bytes: <<1,2,3>>
<<4,5>>
3> Parse(8, <<1,2,3,4,5>>).
Binary shorter than 8
<<1,2,3,4,5>>
Note that this version is a little safer, in that we avoid a crash in the case that Length is longer than the binary. This is yet another good reason why maybe we can't do that match in the function head.
Try with below code:
{<<A:Length/binary, Rest/binary>>, _} = {_, Length} = {<<1,2,3,4,5>>, 3}.
This question is mentioned a bit in EEP-52:
Any variables used in the expression must have been previously bound, or become bound in the same binary pattern as the expression. That is, the following example is illegal:
illegal_example2(N, <<X:N,T/binary>>) ->
{X,T}.
And explained a bit more in the following e-mail: http://erlang.org/pipermail/eeps/2020-January/000636.html
Illegal. With one exception, matching is not done in a left-to-right
order, but all variables in the pattern will be bound at the same
time. That means that the variables must be bound before the match
starts. For maps, that means that the variables referenced in key
expressions must be bound before the case (or receive) that matches
the map. In a function head, all map keys must be literals.
The exception to this general rule is that within a binary pattern,
the segments are matched from left to right, and a variable bound in a
previous segment can be used in the size expression for a segment
later in the binary pattern.
Also one of the members of OTP team mentioned that they made a prototype that can do that, but it was never finished http://erlang.org/pipermail/erlang-questions/2020-May/099538.html
We actually tried to make your example legal. The transformation of
the code that we did was not to rewrite to guards, but to match
arguments or parts of argument in the right order so that variables
that input variables would be bound before being used. (We would do a
topological sort to find the correct order.) For your example, the
transformation would look similar to this:
legal_example(Key, Map) ->
case Map of
#{Key := Value} -> Value;
_ -> error(function_clause, [Key, Map])
end.
In the prototype implementation, the compiler could compile the
following example:
convoluted(Ref,
#{ node(Ref) := NodeId, Loop := universal_answer},
[{NodeId, Size} | T],
<<Int:(Size*8+length(T)),Loop>>) when is_reference(Ref) ->
Int.
Things started to fall apart when variables are repeated. Repeated
variables in patterns already have a meaning in Erlang (they should be
the same), so it become tricky to understand to distinguish between
variables being bound or variables being used a binary size or map
key. Here is an example that the prototype couldn't handle:
foo(#{K := K}, K) -> ok.
A human can see that it should be transformed similar to this:
foo(Map, K) -> case Map of
{K := V} when K =:= V -> ok end.
Here are few other examples that should work but the prototype would
refuse to compile (often emitting an incomprehensible error message):
bin2(<<Sz:8,X:Sz>>, <<Y:Sz>>) -> {X,Y}.
repeated_vars(#{K := #{K := K}}, K) -> K.
match_map_bs(#{K1 := {bin,<<Int:Sz>>}, K2 := <<Sz:8>>}, {K1,K2}) ->
Int.
Another problem was when example was correctly rejected, the error
message would be confusing.
Because much more work would clearly be needed, we have shelved the
idea for now. Personally, I am not sure that the idea is sound in the
first place. But I am sure of one thing: the implementation would be
very complicated.
UPD: latest news from 2020-05-14
I am just starting to learn F#, and impressed by the type inference I thought I would try a function that gets the first record from a table (using query expressions, Linq style):
let getfirst data =
let result = query { for n in data do take 1 }
result |> Seq.head
This works, the type is IQueryable<'a> -> 'a.
But why doesn't this version work?
let getfirst2 data =
query { for n in data do head }
Shouldn't for n in data do head give a scalar 'a just like last time? Can someone explain why the second version doesn't work, and how to make it work without using Seq.head?
The reason is that the query builder has a somewhat hacky overloaded Run method for running queries, with the following overloads:
QueryBuilder.Run : Quotations.Expr<'t> -> 't
QueryBuilder.Run : Quotations.Expr<Linq.QuerySource<'t, IEnumerable>> -> seq<'t>
QueryBuilder.Run : Quotations.Expr<Linq.QuerySource<'t, IQueryable>> -> IQueryable<'t>
In your case, any of the overloads could apply, given a suitable type for data (though QuerySource<_,_> is a type which isn't ever meant to be used by user code, so two of the overloads are quite unlikely). Unfortunately, due to the strange way these overloads are defined (the first and second are actually extension methods defined in separate modules), the third one wins the overload resolution battle.
I don't know why, but when you hover over the data argument in getfirst2 you see it's of type System.Linq.IQueryable<Linq.QuerySource<'a, System.Linq.IQueryable>> when it really should be System.Linq.IQueryable<'a>.
You can "fix" it by adding type annotations:
open System.Linq
let getfirst2 (data : IQueryable<'a>) : 'a = query {
for item in data do
head
}
Then it works like you have expected:
[1 .. 10]
|> System.Linq.Queryable.AsQueryable
|> getfirst2
|> printfn "%d" // Prints 1.
Maybe someone else can shed some light on why the compiler infers the types it does.
my question now is :
I have the variavle M which contains : 37.5 (as you see is integer)
I want to convert M in order to be string "37.5"
so 37.5 should became "37.5"
I try with function :
M2=integer_to_list(M)
but when I execute this function it displays this error :
** exception error: bad argument
in function integer_to_list/1
called as integer_to_list(37.5)
integer_to_list wont work in that situation because 37.5 is a float and not an integer. Erlang does have float_to_list, but the output is usually pretty unusable.
Instead, I would recommend looking into mochiweb project for pretty conversion of floats to lists. In particular, the mochinum module:
> M = 37.5,
> mochinum:digits(M).
"37.5"
#chops has a great answer, IMO (using mochinum:digits/1), but you might also get something out of looking at the io_lib module. For example:
8> io_lib:format("~.2f",[37.5]).
["37.50"]
9> io_lib:format("~.1f",[37.5]).
["37.5"]
I realize this might not be exactly what you are looking for, and in this case I think looking at/using the mochinum module is an efficient way to go, but io_lib is often overlooked and provides a really useful set of functions for formatting lists / strings
Is there a generic way, given a complex object in Erlang, to come up with a valid function declaration for it besides eyeballing it? I'm maintaining some code previously written by someone who was a big fan of giant structures, and it's proving to be error prone doing it manually.
I don't need to iterate the whole thing, just grab the top level, per se.
For example, I'm working on this right now -
[[["SIP",47,"2",46,"0"],32,"407",32,"Proxy Authentication Required","\r\n"],
[{'Via',
[{'via-parm',
{'sent-protocol',"SIP","2.0","UDP"},
{'sent-by',"172.20.10.5","5060"},
[{'via-branch',"z9hG4bKb561e4f03a40c4439ba375b2ac3c9f91.0"}]}]},
{'Via',
[{'via-parm',
{'sent-protocol',"SIP","2.0","UDP"},
{'sent-by',"172.20.10.15","5060"},
[{'via-branch',"12dee0b2f48309f40b7857b9c73be9ac"}]}]},
{'From',
{'from-spec',
{'name-addr',
[[]],
{'SIP-URI',
[{userinfo,{user,"003018CFE4EF"},[]}],
{hostport,"172.20.10.11",[]},
{'uri-parameters',[]},
[]}},
[{tag,"b7226ffa86c46af7bf6e32969ad16940"}]}},
{'To',
{'name-addr',
[[]],
{'SIP-URI',
[{userinfo,{user,"3966"},[]}],
{hostport,"172.20.10.11",[]},
{'uri-parameters',[]},
[]}},
[{tag,"a830c764"}]},
{'Call-ID',"90df0e4968c9a4545a009b1adf268605#172.20.10.15"},
{'CSeq',1358286,"SUBSCRIBE"},
["date",'HCOLON',
["Mon",44,32,["13",32,"Jun",32,"2011"],32,["17",58,"03",58,"55"],32,"GMT"]],
{'Contact',
[[{'name-addr',
[[]],
{'SIP-URI',
[{userinfo,{user,"3ComCallProcessor"},[]}],
{hostport,"172.20.10.11",[]},
{'uri-parameters',[]},
[]}},
[]],
[]]},
["expires",'HCOLON',3600],
["user-agent",'HCOLON',
["3Com",[]],
[['LWS',["VCX",[]]],
['LWS',["7210",[]]],
['LWS',["IP",[]]],
['LWS',["CallProcessor",[['SLASH',"v10.0.8"]]]]]],
["proxy-authenticate",'HCOLON',
["Digest",'LWS',
["realm",'EQUAL',['SWS',34,"3Com",34]],
[['COMMA',["domain",'EQUAL',['SWS',34,"3Com",34]]],
['COMMA',
["nonce",'EQUAL',
['SWS',34,"btbvbsbzbBbAbwbybvbxbCbtbzbubqbubsbqbtbsbqbtbxbCbxbsbybs",
34]]],
['COMMA',["stale",'EQUAL',"FALSE"]],
['COMMA',["algorithm",'EQUAL',"MD5"]]]]],
{'Content-Length',0}],
"\r\n",
["\n"]]
Maybe https://github.com/etrepum/kvc
I noticed your clarifying comment. I'd prefer to add a comment myself, but don't have enough karma. Anyway, the trick I use for that is to experiment in the shell. I'll iterate a pattern against a sample data structure until I've found the simplest form. You can use the _ match-all variable. I use an erlang shell inside an emacs shell window.
First, bind a sample to a variable:
A = [{a,b},[{c,d}, {e,f}]].
Now set the original structure against the variable:
[{a,b},[{c,d},{e,f}]] = A.
If you hit enter, you'll see they match. Hit alt-p (forget what emacs calls alt, but it's alt on my keyboard) to bring back the previous line. Replace some tuple or list item with an underscore:
[_,[{c,d},{e,f}]].
Hit enter to make sure you did it right and they still match. This example is trivial, but for deeply nested, multiline structures it's trickier, so it's handy to be able to just quickly match to test. Sometimes you'll want to try to guess at whole huge swaths, like using an underscore to match a tuple list inside a tuple that's the third element of a list. If you place it right, you can match the whole thing at once, but it's easy to misread it.
Anyway, repeat to explore the essential shape of the structure and place real variables where you want to pull out values:
[_, [_, _]] = A.
[_, _] = A.
[_, MyTupleList] = A. %% let's grab this tuple list
[{MyAtom,b}, [{c,d}, MyTuple]] = A. %% or maybe we want this atom and tuple
That's how I efficiently dissect and pattern match complex data structures.
However, I don't know what you're doing. I'd be inclined to have a wrapper function that uses KVC to pull out exactly what you need and then distributes to helper functions from there for each type of structure.
If I understand you correctly you want to pattern match some large datastructures of unknown formatting.
Example:
Input: {a, b} {a,b,c,d} {a,[],{},{b,c}}
function({A, B}) -> do_something;
function({A, B, C, D}) when is_atom(B) -> do_something_else;
function({A, B, C, D}) when is_list(B) -> more_doing.
The generic answer is of course that it is undecidable from just data to know how to categorize that data.
First you should probably be aware of iolists. They are created by functions such as io_lib:format/2 and in many other places in the code.
One example is that
[["SIP",47,"2",46,"0"],32,"407",32,"Proxy Authentication Required","\r\n"]
will print as
SIP/2.0 407 Proxy Authentication Required
So, I'd start with flattening all those lists, using a function such as
flatten_io(List) when is_list(List) ->
Flat = lists:map(fun flatten_io/1, List),
maybe_flatten(Flat);
flatten_io(Tuple) when is_tuple(Tuple) ->
list_to_tuple([flatten_io(Element) || Element <- tuple_to_list(Tuple)];
flatten_io(Other) -> Other.
maybe_flatten(L) when is_list(L) ->
case lists:all(fun(Ch) when Ch > 0 andalso Ch < 256 -> true;
(List) when is_list(List) ->
lists:all(fun(X) -> X > 0 andalso X < 256 end, List);
(_) -> false
end, L) of
true -> lists:flatten(L);
false -> L
end.
(Caveat: completely untested and quite inefficient. Will also crash for inproper lists, but you shouldn't have those in your data structures anyway.)
On second thought, I can't help you. Any data structure that uses the atom 'COMMA' for a comma in a string should be taken out and shot.
You should be able to flatten those things as well and start to get a view of what you are looking at.
I know that this is not a complete answer. Hope it helps.
Its hard to recommend something for handling this.
Transforming all the structures in a more sane and also more minimal format looks like its worth it. This depends mainly on the similarities in these structures.
Rather than having a special function for each of the 100 there must be some automatic reformatting that can be done, maybe even put the parts in records.
Once you have records its much easier to write functions for it since you don't need to know the actual number of elements in the record. More important: your code won't break when the number of elements changes.
To summarize: make a barrier between your code and the insanity of these structures by somehow sanitizing them by the most generic code possible. It will be probably a mix of generic reformatting with structure speicific stuff.
As an example already visible in this struct: the 'name-addr' tuples look like they have a uniform structure. So you can recurse over your structures (over all elements of tuples and lists) and match for "things" that have a common structure like 'name-addr' and replace these with nice records.
In order to help you eyeballing you can write yourself helper functions along this example:
eyeball(List) when is_list(List) ->
io:format("List with length ~b\n", [length(List)]);
eyeball(Tuple) when is_tuple(Tuple) ->
io:format("Tuple with ~b elements\n", [tuple_size(Tuple)]).
So you would get output like this:
2> eyeball({a,b,c}).
Tuple with 3 elements
ok
3> eyeball([a,b,c]).
List with length 3
ok
expansion of this in a useful tool for your use is left as an exercise. You could handle multiple levels by recursing over the elements and indenting the output.
Use pattern matching and functions that work on lists to extract only what you need.
Look at http://www.erlang.org/doc/man/lists.html:
keyfind, keyreplace, L = [H|T], ...
So I've been using Erlang for the last eight hours, and I've spent two of those banging my head against the keyboard trying to figure out the exception error my console keeps returning.
I'm writing a dice program to learn erlang. I want it to be able to call from the console through the erlang interpreter. The program accepts a number of dice, and is supposed to generate a list of values. Each value is supposed to be between one and six.
I won't bore you with the dozens of individual micro-changes I made to try and fix the problem (random engineering) but I'll post my code and the error.
The Source:
-module(dice2).
-export([d6/1]).
d6(1) ->
random:uniform(6);
d6(Numdice) ->
Result = [],
d6(Numdice, [Result]).
d6(0, [Finalresult]) ->
{ok, [Finalresult]};
d6(Numdice, [Result]) ->
d6(Numdice - 1, [random:uniform(6) | Result]).
When I run the program from my console like so...
dice2:d6(1).
...I get a random number between one and six like expected.
However when I run the same function with any number higher than one as an argument I get the following exception...
**exception error: no function clause matching dice2:d6(1, [4|3])
... I know I I don't have a function with matching arguments but I don't know how to write a function with variable arguments, and a variable number of arguments.
I tried modifying the function in question like so....
d6(Numdice, [Result]) ->
Newresult = [random:uniform(6) | Result],
d6(Numdice - 1, Newresult).
... but I got essentially the same error. Anyone know what is going on here?
This is basically a type error. When Result is a list, [Result] is a list with one element. E.g., if your function worked, it would always return a list with one element: Finalresult.
This is what happens (using ==> for "reduces to"):
d6(2) ==> %% Result == []
d6(2, [[]]) ==> %% Result == [], let's say random:uniform(6) gives us 3
d6(1, [3]) ==> %% Result == 3, let's say random:uniform(6) gives us 4
d6(0, [4|3]) ==> %% fails, since [Result] can only match one-element lists
Presumably, you don't want [[]] in the first call, and you don't want Result to be 3 in the third call. So this should fix it:
d6(Numdice) -> Result = [], d6(Numdice, Result). %% or just d6(Numdice, []).
d6(0, Finalresult) -> {ok, Finalresult};
d6(Numdice, Result) -> d6(Numdice - 1, [random:uniform(6) | Result]).
Lesson: if a language is dynamically typed, this doesn't mean you can avoid getting the types correct. On the contrary, it means that the compiler won't help you in doing this as much as it could.