Erlang: Faster way to sum bytes in a binary - erlang

Below is simple implemenation that adds bytes of a binary. It is slow according to eprof (takes about 10% of total time - mostly because of many calls to binary:part/3).
How can this be optimzed ?
calc_checksum(Packet) when is_binary(Packet)->
calc_checksum(Packet, 0).
calc_checksum(<<>>, Acc) ->
Acc band 16#FFFF;
calc_checksum(Packet, Acc) when is_binary(Packet) ->
W = binary:decode_unsigned(binary:part(Packet, 0, 2), little),
NextAcc = Acc + W,
NextBytes = binary:part(Packet, byte_size(Packet), -(byte_size(Packet)-2)),
calc_checksum(NextBytes, NextAcc).

A more elegant solution is:
calc_checksum(<<W:16/little,Rest/bytes>>, Acc0) ->
Acc1 = Acc0 + W,
calc_checksum(Rest, Acc1);
calc_checksum(<<>>, Acc) -> Acc band 16#FFFF.
This code will generate an error if the binary contains an odd number of bytes. Using pattern matching usually gives better more elegant code.

Using pattern matching instead of calling functions in binary seems to double the speed in the pseudo-benchmark I tried in the shell. Something like this:
calc_checksum(Packet, Acc) when is_binary(Packet) ->
<<W:16/little, NextBytes/binary>> = Packet,
NextAcc = Acc + W,
calc_checksum(NextBytes, NextAcc).
(I might be wrong, but you should get equivalent results if you set NextAcc to (Acc + W) band 16#FFFF, which should avoid bignums if you run this on really big binaries.)

If you process several values in once you can even speed up little bit more than Robert's solution:
calc_checksum(<<W1:16/little, W2:16/little, W3:16/little, W4:16/little, Rest/bytes>>, Acc)->
calc_checksum(Rest, Acc+W1+W2+W3+W4);
calc_checksum(<<W:16/little,Rest/bytes>>, Acc) ->
calc_checksum(Rest, Acc+W);
calc_checksum(<<>>, Acc) -> Acc band 16#FFFF.

Related

Erlang; list comprehension without duplicates

I am doing somthing horrible but I don't know how to make it better.
I am forming all pairwise sums of the elements of a List called SomeList, but I don't want to see duplicates ( I guess I want "all possible pairwise sums" ):
sets:to_list(sets:from_list([A+B || A <- SomeList, B <- SomeList]))
SomeList does NOT contain duplicates.
This works, but is horribly inefficient, because the original list before the set conversion is GIGANTIC.
Is there a better way to do this?
You could simply use lists:usort/1
lists:usort([X+Y || X <- L, Y <- L]).
if the chance to have duplicates is very high, then you can generate the sum using 2 loops and store the sum in an ets set (or using map, I didn't check the performance of both).
7> Inloop = fun Inloop(_,[],_) -> ok; Inloop(Store,[H|T],X) -> ets:insert(Store,{X+H}), Inloop(Store,T,X) end.
#Fun<erl_eval.42.54118792>
8> Outloop = fun Outloop(Store,[],_) -> ok; Outloop(Store,[H|T],List) -> Inloop(Store,List,H), Outloop(Store,T,List) end.
#Fun<erl_eval.42.54118792>
9> Makesum = fun(L) -> S = ets:new(temp,[set]), Outloop(S,L,L), R =ets:foldl(fun({X},Acc) -> [X|Acc] end,[],S), ets:delete(S), R end.
#Fun<erl_eval.6.54118792>
10> Makesum(lists:seq(1,10)).
[15,13,8,11,20,14,16,12,7,3,10,9,19,18,4,17,6,2,5]
11> lists:sort(Makesum(lists:seq(1,10))).
[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
12>
This module will allow you to compare times of execution when using list comprehension, sets or ets. You can of course add additional functions to this comparison:
-module(pairwise).
-export([start/2]).
start(Type, X) ->
L = lists:seq(1, X),
timer:tc(fun do/2, [Type, L]).
do(compr, L) ->
sets:to_list(sets:from_list([A+B || A <- L, B <- L]));
do(set, L) ->
F = fun(Sum, Set) -> sets:add_element(Sum, Set) end,
R = fun(Set) -> sets:to_list(Set) end,
do(L, L, sets:new(), {F, R});
do(ets, L) ->
F = fun(Sum, Tab) -> ets:insert(Tab, {Sum}), Tab end,
R = fun(Tab) ->
Fun = fun({X}, Acc) -> [X|Acc] end,
Res = ets:foldl(Fun, [], Tab),
ets:delete(Tab),
Res
end,
do(L, L, ets:new(?MODULE, []), {F, R}).
do([A|AT], [B|BT], S, {F, _} = Funs) -> do([A|AT], BT, F(A+B, S), Funs);
do([_AT], [], S, {_, R}) -> R(S);
do([_A|AT], [], S, Funs) -> do(AT, AT, S, Funs).
Results:
36> {_, Res1} = pairwise:start(compr, 20).
{282,
[16,32,3,19,35,6,22,38,9,25,12,28,15,31,2,18,34,5,21,37,8,
24,40,11,27,14,30|...]}
37> {_, Res2} = pairwise:start(set, 20).
{155,
[16,32,3,19,35,6,22,38,9,25,12,28,15,31,2,18,34,5,21,37,8,
24,40,11,27,14,30|...]}
38> {_, Res3} = pairwise:start(ets, 20).
{96,
[15,25,13,8,21,24,40,11,26,20,14,28,23,16,12,39,34,36,7,32,
35,3,33,10,9,19,18|...]}
39> R1=lists:usort(Res1), R2=lists:usort(Res2), R3=lists:usort(Res3).
[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,
24,25,26,27,28,29,30|...]
40> R1 = R2 = R3.
[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,
24,25,26,27,28,29,30|...]
The last line is to compare that all functions return the same result but sorted differently.
First number in each resulted tuple is the time of execution as returned from timer:tc(fun do/2, [Type, L]).. In this example it's 282 for list comprehension, 155 for sets and 96 for ets.
An effective way is to use foldl instead of lists comprehension, because in this case you nedd a state on each step
sets:to_list(
lists:foldl(fun(A, S1) ->
lists:foldl(fun(B, S2) ->
sets:add_element(A+B, S2)
end, S1, SomeListA)
end, sets:new(), SomeListB)).
This solution keeps it relatively fast and makes use of as much pre-written library code as possible.
Note that I use lists:zip/2 here rather than numeric +, only to illustrate that this approach works for any kind of non-repeating permutation of a unique list. You may only care about arithmetic, but if you want more, this can do it.
-export([permute_unique/1]).
permute_unique([]) ->
[];
permute_unique([A|Ab]) ->
lists:zip(lists:duplicate(length(Ab)+1, A), [A|Ab])
++
permute_unique(Ab).
%to sum integers, replace the lists:zip... line with
% [B+C || {B,C} <- lists:zip(lists:duplicate(length(Ab)+1, A), [A|Ab])]
%to perform normal arithmetic and yield a numeric value for each element
I am not sure what you consider gigantic - you will end up with N*(N+1)/2 total elements in the permuted list for a unique list of N original elements, so this gets big really fast.
I did some basic performance testing of this, using an Intel (Haswell) Core i7 # 4GHz with 32GB of memory, running Erlang/OTP 17 64-bit.
5001 elements in the list took between 2 and 5 seconds according to timer:tc/1.
10001 elements in the list took between 15 and 17 seconds, and required about 9GB of memory. This generates a list of 50,015,001 elements.
15001 elements in the list took between 21 and 25 seconds, and required about 19GB of memory.
20001 elements in the list took 49 seconds in one run, and peaked at about 30GB of memory, with about 200 million elements in the result. That is the limit of what I can test.

How to get an random integer with limit length?

I want to create a function get_id(max_length). At first want to math:pow/2, but it return float data type. It seems not a good idea.
with code as follows, but only support max length=20, as it is hardcoded, any good idea?
seed()->
{M_a,M_b,M_c} = now(),
random:seed(M_a,M_b,M_c),
ok.
get_id(1)-> random:uniform(1);
get_id(2) -> random:uniform(10);
get_id(3) -> random:uniform(100);
get_id(4) -> random:uniform(1000);
get_id(5) -> random:uniform(10000);
get_id(6) -> random:uniform(100000);
get_id(7) -> random:uniform(1000000);
get_id(8) -> random:uniform(10000000);
get_id(9) -> random:uniform(100000000);
get_id(10) -> random:uniform(1000000000);
get_id(11) -> random:uniform(10000000000);
get_id(12) -> random:uniform(100000000000);
get_id(13) -> random:uniform(1000000000000);
get_id(14) -> random:uniform(10000000000000);
get_id(15) -> random:uniform(100000000000000);
get_id(16) -> random:uniform(1000000000000000);
get_id(17) -> random:uniform(10000000000000000);
get_id(18) -> random:uniform(100000000000000000);
get_id(19) -> random:uniform(1000000000000000000);
get_id(20) -> random:uniform(10000000000000000000).
Your approach, unfortunately, doesn't work. Indeed, while random:uniform/1 accepts any positive integer as its argument, it does not deliver a random integer uniformly distributed between 1 and N for very large values of N (in spite of what documentation claims).
The reason is that random:uniform/1 is actually truncating the product of its argument by the value of random:uniform/0 (and adding 1 for [1-N] range instead of [0-(N-1)]).
See source code: https://github.com/erlang/otp/blob/maint/lib/stdlib/src/random.erl#L112
And floats are IEEE 754 doubles with 53 bits mantissa, which means that get_id/1 will not return all possible values for input from 17 to 20 (with 16 or more digits).
random:uniform/0,1 is known as a poor random generator, mostly suitable if you want to generate reproductible pseudo-random series (a given seed value will always generate the same series). For this reason, I would suggest using crypto:rand_uniform/2.
A simple solution would be to compute 10^(N-1) using integer arithmetics (to avoid the 53 bits mantissa issue) and then call crypto:rand_uniform/2. You can perform this with a naive recursive implementation (pow1/1 below), or use binary exponentiation (pow2/1 below).
-define(BASE, 10).
-spec pow1(non_neg_integer()) -> pos_integer().
pow1(N) when N >= 0 ->
pow1(N, 1).
pow1(0, Acc) -> Acc;
pow1(N, Acc) ->
pow1(N - 1, Acc * ?BASE).
-spec pow2(non_neg_integer()) -> pos_integer().
pow2(N) when N >= 0 ->
pow2(?BASE, N, 1).
pow2(_X, 0, Acc) ->
Acc;
pow2(X, N, Acc) when N rem 2 =:= 0 ->
pow2(X * X, N div 2, Acc);
pow2(X, N, Acc) ->
pow2(X * X, N div 2, Acc * X).
Your function could simply be written as:
-spec get_id2(pos_integer()) -> non_neg_integer().
get_id2(N) ->
1 + crypto:rand_uniform(0, pow2(N - 1)).
Alternatively, you could use a combination of uniform random variables, one per digit (while the sum of two random uniform variables is generally not a uniform random variable, it is if combined like this) or for several digits in the case of the binary exponentiation.
With the naive exponentiation:
-spec get_id3(pos_integer()) -> pos_integer().
get_id3(N) when N > 0 ->
get_id3(N - 1, 0).
get_id3(0, Acc) -> 1 + Acc;
get_id3(N, Acc) ->
Acc1 = crypto:rand_uniform(0, ?BASE) + (Acc * ?BASE),
get_id3(N - 1, Acc1).
With the binary exponentiation:
-spec get_id4(pos_integer()) -> pos_integer().
get_id4(N) when N > 0 ->
get_id4(?BASE, N - 1, 0).
get_id4(_X, 0, Acc) ->
1 + Acc;
get_id4(X, N, Acc) when N rem 2 =:= 0 ->
get_id4(X * X, N div 2, Acc);
get_id4(X, N, Acc) ->
Acc1 = crypto:rand_uniform(0, X) + (Acc * X),
get_id4(X * X, N div 2, Acc1).
Why not use trunc/1 to cast the floats returned by math:pow/2 to integers? http://www.erlang.org/doc/man/erlang.html#trunc-1
like in any language, you can get a power of 2 by shifting left the number 1:
1> 1 bsl 3.
8
2> 1 bsl 8.
256
3> 1 bsl 852.
30030067315218800919884630782037027445247038374198014146711597563050526250476926831789640794321325523394216076738821850476730762665208973047045843626559620640158907690363610309346513399556581649279919071671610504617321356178738468477058455548958390664298496
4>
As you can see, the size of integer is not limited in erlang. It is both good and bad since small integer (that is integer represented as a single worg like in most languages) are limited depending on the architecture:
On 32-bit architectures: -134217729 < i < 134217728 (28 bits)
On 64-bit architectures: -576460752303423489 < i < 576460752303423488 (60 bits)
for bigger integer, another representation is used: big integer, that takes more space in memory and take longer to compute.

How to collect frequencies of characters using a list of tuples {char,freq} in Erlang

I am supposed to collect frequencies of characters.
freq(Sample) -> freq(Sample,[]).
freq([],Freq) ->
Freq;
freq([Char|Rest],Freq)->
freq(Rest,[{Char,1}|Freq]).
This function does not work in the right way. If the input is "foo", then the output will be
[{f,1},{o,1},{o,1}].
But I wished to have the output like
[{f,1},{o,2}].
I can't manage to modify element in a tulpe. Can anyone help me out of this and show me how it can be fixed?
a one line solution :o)
% generate a random list
L = [random:uniform(26)+$a-1 || _ <- lists:seq(1,1000)].
% collect frequency
lists:foldl(fun(X,[{[X],I}|Q]) -> [{[X],I+1}|Q] ; (X,Acc) -> [{[X],1}|Acc] end , [], lists:sort(L)).
in action
1> lists:foldl(fun(X,[{[X],I}|Q]) -> [{[X],I+1}|Q] ; (X,Acc) -> [{[X],1}|Acc] end , [], lists:sort("foo")).
[{"o",2},{"f",1}]
quite fast with short list, but the execution time increase a lot with long list (on my PC, it needs 6.5s for a 1 000 000 character text) .
in comparison, with the same 1 000 000 character text Ricardo solution needs 5 sec
I will try another version using ets.
By far the easiest way is to use an orddict to store the value as it already comes with an update_counter function and returns the value in a (sorted) list.
freq(Text) ->
lists:foldl(fun (C, D) -> orddict:update_counter(C, 1, D) end, orddict:new(), Text).
Try with something like this:
freq(Text) ->
CharsDictionary = lists:foldl(fun(Char, Acc) -> dict:update_counter(Char, 1, Acc) end, dict:new(), Text),
dict:fold(fun(Char, Frequency, Acc) -> [{Char, Frequency} | Acc] end, [], CharsDictionary).
The first line creates a dictionary that uses the char as key and the frequency as value (dict:update_counter).
The second line converts the dictionary in the list that you need.
Using pattern matching and proplists.
-module(freq).
-export([char_freq/1]).
-spec char_freq(string()) -> [tuple()].
char_freq(L) -> char_freq(L, []).
char_freq([], PL) -> PL;
char_freq([H|T], PL) ->
case proplists:get_value([H], PL) of
undefined ->
char_freq(T, [{[H],1}|PL]);
N ->
L = proplists:delete([H], PL),
char_freq(T, [{[H],N+1}|L])
end.
Test
1> freq:char_freq("abacabz").
[{"z",1},{"b",2},{"a",3},{"c",1}]
L = [list_to_atom(X) || X <- Str].
D = lists:foldl(fun({Char, _}, Acc) -> dict:update_counter(Char, 1, Acc) end, dict:new(), L).
dict:to_list(D).

Fibonacci Matrix

For calculating a fibonacci sequence in O(logn) we use matrix exponential since the term
fn = fn-1 + fn-2 is linear but what is the matrix required if we want to find nth term of
fn = fn-1 + fn-2 + a0 + a1*n + a2*n^2 + ... an*n^n
which is a dependent on polynomial???
Here a0,a1,... an are constants
Look here for implementation in Erlang which uses formula
. It shows nice linear resulting behavior because in O(M(n) log n) part M(n) is exponential for big numbers. It calculates fib of one million in 2s where result has 208988 digits. The trick is that you can compute exponentiation in O(log n) multiplications using (tail) recursive formula (tail means with O(1) space when used proper compiler or rewrite to cycle):
% compute X^N
power(X, N) when is_integer(N), N >= 0 ->
power(N, X, 1).
power(0, _, Acc) ->
Acc;
power(N, X, Acc) ->
if N rem 2 =:= 1 ->
power(N - 1, X, Acc * X);
true ->
power(N div 2, X * X, Acc)
end.
where X and Acc you substitute with matrices. X will be initiated with and Acc with identity I equals to .

how to convert a large number exprimed on several bytes?

If a number is exprimed on 4 bytes, from LSB to MSB, how to convert it in integer ?
example:
<<77,0,0,0>> shall give 77
but
<<0,1,0,0>> shall give 256
Let S = <<0,1,0,0>>,
<<L1,L2,L3,L4>> = S,
L = L1*1 + L2*256 + L3*65536 + L4*16777216,
But it's not elegant ...
The bit syntax in Erlang does this in a very straightforward way:
<<A:32/little>> = <<0,1,0,0>>,
A.
% A = 256
or as a function:
decode(<<Int:32/little>>) -> Int.
% decode(<<0,1,0,0>>) =:= 256.
EDIT (this is the correct answer, and sorry for discovering it late...)
> binary:decode_unsigned(<<0,1,0,0>>,little).
256
The easier way would be something like:
decode_my_binary( <<A,B,C,D>> ) ->
A + B*256 + C*65536 + D*16777216.
EDIT:
As per your edit, if you find this one not very elegant, you can try other approaches. Still I think the above is the correct way of doing it. You can write a recursive function (not tested, but you get the idea):
decode( B ) -> decode(binary_to_list(B), 0, 1).
decode( [], R, _ ) -> R;
decode( [H|T], R, F) ->
decode(T, R + H*F, F*256).
but this is clearly slower. Another possibility is to have the list of the binary digits and the list of multipliers and then fold it:
lists:sum(lists:zipwith( fun(X,Y) -> X*Y end,
binary_to_list(B), [ math:pow(256,X) || X <- [0,1,2,3] ])).
Or if you want a variable number of digits:
fun(Digits) ->
lists:sum(lists:zipwith( fun(X,Y) -> X*Y end,
binary_to_list(B), [ math:pow(256,X) || X <- lists:seq(0,Digits-1])).
where Digits tell you the digit number.

Resources