Related
a = [1, 2, 3],
b = [4, 5, 6],
c = [1, 7, 8],
has_common_element(a, b) is false since there are no common elements between the two arrays.
has_common_element(a, c) is true since there is at least one common element (1) between the two arrays.
I can use lists:member to check whether a single element is in the list. How do I implement has_common_element in Erlang to check whether two lists have a common element?
Create sets from the lists and use sets:is_disjoint:
has_common_member(L1, L2) ->
not(sets:is_disjoint(sets:from_list(L1), sets:from_list(L2))).
You can use list comprehension like below:
test(A, B)->
[] /= [X || X <- A, Y <- B, X == Y].
the result in shell:
1> A = [1, 2, 3].
[1,2,3]
2> B = [4, 5, 6].
[4,5,6]
3> C = [1, 7, 8].
[1,7,8]
4> foo:test(A, B).
false
5> foo:test(A, C).
true
You can make fast solution using erlang:--/2 operator:
has_common_element(A, B) when length(B) > length(A) ->
has_common_element(B, A); % not necessary optimization but helps for very uneven sets
has_common_element(A, B) ->
A =/= A -- B.
It will be O(N*M) in older versions but since OTP21 I think it becomes O((N+M)*logM) (I can't find relevant release note just now). Benchmark for yourself in the version you are using and choose properly.
Anyway You can make still pretty fast O((N+M)*logM) solution using maps:
has_common_element(A, B) when length(B) > length(A) ->
has_common_element(B, A); % not necessary optimization but helps for very uneven sets
has_common_element(A, B) ->
M = maps:from_list([{X,[]} || X <- B]),
any_in_map(M, A).
any_in_map(_, []) -> false;
any_in_map(M, [H|T]) ->
maps:is_key(H, M) orelse any_in_map(M, T).
And there is a nasty slightly faster version of previous using process_dictionary. You can use it directly when you have good control over list elements and process_dictionary doesn't bother you (usually not good idea in OTP process).
has_common_element_dirty(A, B) when length(B) > length(A) ->
has_common_element_dirty(B, A); % not necessary optimization but helps for very uneven sets
has_common_element_dirty(A, B) ->
[put(X, []) || X <- B],
any_in_dict(A).
any_in_dict([]) -> false;
any_in_dict([H|T]) ->
[] =:= get(H) orelse any_in_dict(T).
You can wrap it in the linked process and make it safe:
has_common_element(A, B) ->
Self = self(),
Ref = make_ref(),
Pid = spawn_link(fun() -> Self ! {Ref, has_common_element_dirty(A, B)} end),
receive {Ref, Result} -> Result end.
Benchmark for your application and list sizes and choose accordingly because all those solutions have different GC and raw performance characteristics. Usually, the first one should be sufficient.
Erlang/OTP 22 [erts-10.4] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Eshell V10.4 (abort with ^G)
1> Size=100000, A = [rand:uniform(Size*10000) || _ <- lists:duplicate(Size, 0)], B = [rand:uniform(Size*10000) || _ <- lists:duplicate(Size, 0)], C = [rand:uniform(Size*10000)+Size*10000 || _ <- lists:duplicate(Size, 0)], ok.
ok
2> timer:tc(fun() -> A =/= A -- B end).
{77943,true}
3> timer:tc(fun() -> A =/= A -- C end).
{44325,false}
It's less than 100ms for comparing two 100k lists. It's on par or better than the best (s2) from Pascal's solutions. Note OTP22 version.
Both #bxdoan and #choroba solutions work and are very concise. You do not give information about the size of the lists and their content, and it may be important.
The solution with list comprehension tests a list which size is the cross product of the 2 input list and thus gets very slow with big input lists. The solution with sets grows less faster. I propose you another one, less concise, but which behaves better for longer lists. I also add a solution using lists:member (not efficient) and another using maps (similar to sort).
-module (inter).
-export ([test/2,s1/2,s2/2,s3/2,s4/2,s5/2]).
test(Method,Size) ->
A = [rand:uniform(Size*10000) || _ <- lists:duplicate(Size, 0)],
% B may contain common element with A
B = [rand:uniform(Size*10000) || _ <- lists:duplicate(Size, 0)],
% C is by construction disjoint with A
C = [rand:uniform(Size*10000)+Size*10000 || _ <- lists:duplicate(Size, 0)],
io:format("test algorithm ~s~n",[method(Method)]),
{test(Method,A,B),test(Method,A,C)}.
test(M,L1,L2) ->
timer:tc(?MODULE, M, [L1,L2]).
method(s1) ->
"List comprehension";
method(s2) ->
"List sorted";
method(s3) ->
"List to set";
method(s4) ->
"List to map";
method(s5) ->
"lists:member".
s1(L1,L2) ->
[] =/= [X || X <- L1, Y <- L2, X =:= Y].
s2(L1,L2) ->
s2_help(lists:sort(L1),lists:sort(L2)).
s2_help([],_) -> false;
s2_help(_,[]) -> false;
s2_help([_A|_],[_A|_]) -> true;
s2_help([H1|T1],[H2|T2]) when H1 > H2 -> s2_help([H1|T1],T2);
s2_help(L1,L2) -> s2_help(tl(L1),L2).
s3(L1,L2) ->
not(sets:is_disjoint(sets:from_list(L1), sets:from_list(L2))).
s4(L1,L2) ->
s4_help(lists:foldl(fun(X,Acc) -> maps:put(X, 0, Acc) end,#{},L1),L2).
s4_help(_,[]) -> false;
s4_help(Map,[H|T]) ->
case maps:is_key(H, Map) of
true -> true;
false -> s4_help(Map,T)
end.
s5([],_) -> false;
s5([H|T],L2) ->
case lists:member(H, L2) of
true -> true;
false -> s5(T,L2)
end.
and then the results:
55> c(inter).
{ok,inter}
56> inter:test(s1,100).
test algorithm List comprehension
{{0,false},{0,false}}
57> inter:test(s2,100).
test algorithm List sorted
{{0,false},{0,false}}
58> inter:test(s3,100).
test algorithm List to set
{{0,false},{0,false}}
59> inter:test(s1,1000).
test algorithm List comprehension
{{16000,true},{0,false}}
60> inter:test(s2,1000).
test algorithm List sorted
{{0,false},{0,false}}
61> inter:test(s3,1000).
test algorithm List to set
{{0,false},{0,false}}
62> inter:test(s1,10000).
test algorithm List comprehension
{{468999,false},{484000,false}}
63> inter:test(s2,10000).
test algorithm List sorted
{{15000,false},{0,false}}
64> inter:test(s3,10000).
test algorithm List to set
{{31000,false},{32000,false}}
65> inter:test(s1,100000).
test algorithm List comprehension
{{48515953,true},{48030953,false}}
66> inter:test(s2,100000).
test algorithm List sorted
{{62000,true},{78000,false}}
67> inter:test(s3,100000).
test algorithm List to set
{{1233999,true},{1296999,false}}
...
69> inter:test(s4,100).
test algorithm List to map
{{0,false},{0,false}}
70> inter:test(s4,1000).
test algorithm List to map
{{0,false},{0,false}}
71> inter:test(s4,10000).
test algorithm List to map
{{0,false},{16000,false}}
72> inter:test(s4,100000).
test algorithm List to map
{{62000,true},{78000,false}}
73> inter:test(s5,100).
test algorithm lists:member
{{0,false},{0,false}}
74> inter:test(s5,1000).
test algorithm lists:member
{{0,false},{0,false}}
75> inter:test(s5,10000).
test algorithm lists:member
{{31000,true},{171999,false}}
76> inter:test(s5,100000).
test algorithm lists:member
{{921000,true},{19031980,false}}
77>
I have a list in erlang containing interger values.
I want to remove values that occur only one time.(Not Duplicates).
Input = [1,3,2,1,2,2]
Output = [1,2,1,2,2]
I am newbie to erlang. I have tried an approach to sorting them first using list:sort() and then removing a member if the member next to it is the same.
I am having trouble trying to iterate the list. It would be great help if you can show me how I can do it.
multiple(L) ->
M = L -- lists:usort(L),
[X || X <- L , lists:member(X,M)].
Use map to count values and then filter values which was not present just once.
-module(test).
-export([remove_unique/1]).
remove_unique(L) ->
Count = lists:foldl(fun count/2, #{}, L),
lists:filter(fun(X) -> maps:get(X, Count) =/= 1 end, L).
count(X, M) ->
maps:put(X, maps:get(X, M, 0) + 1, M).
And test:
1> c(test).
{ok,test}
2> test:remove_unique([1,2,3,3,3,5,5,6,7,7]).
[3,3,3,5,5,7,7]
3> test:remove_unique([1,2,3,3,3,5,5,6,7,8]).
[3,3,3,5,5]
4> test:remove_unique([1,3,2,1,2,2]).
[1,2,1,2,2]
Here's a solution I'd written when first seeing the question when posted, that uses the same logic as #A.Sarid's recursion/pattern matching answer, except that I use a "Last" parameter instead of the count.
-module(only_dupes).
-export([process/1]).
process([]) -> [];
process(L) when is_list(L) ->
[H|T] = lists:sort(L),
lists:sort(process(undefined, H, T, [])).
process(Last, Curr, [], Acc)
when Curr =/= Last ->
Acc;
process(_Last, Curr, [], Acc) ->
[Curr | Acc];
process(Last, Curr, [Next | Rest], Acc)
when Curr =/= Last, Curr =/= Next ->
process(Curr, Next, Rest, Acc);
process(_Last, Curr, [Next | Rest], Acc) ->
process(Curr, Next, Rest, [Curr | Acc]).
One way for iterating a list (that as a result will return a new list) is using recursion and pattern matching.
After you sort your list you want to iterate the list and to check not only that it is different from the next element, but that there was no other equal elements before it. Consider the list [3,3,3,5,5] if you will only check the next element, the last 3 will also be unique and that is incorrect.
Here is a working program, I used a counter to cover the above case as well. See the syntax for using [H|T] for iterating over the list. You may see more cases and read more about it here.
-module(test).
-export([remove_unique/1]).
remove_unique(Input) ->
Sorted = lists:sort(Input),
remove_unique(Sorted, [], 0).
% Base case - checks if element is unique
remove_unique([H|[]],Output,Count) ->
case Count of
0 -> Output;
_Other -> [H|Output]
end;
% Count is 0 - might be unique - check with next element
remove_unique([H1|[H2|T]],Output, 0)->
case (H1 =:= H2) of
true -> remove_unique([H2|T],[H1|Output],1);
false -> remove_unique([H2|T],Output,0)
end;
% Count is > 0 - not unique - proceed adding to list until next value
remove_unique([H1|[H2|T]],Output,Count) ->
case (H1 =:= H2) of
true -> remove_unique([H2|T],[H1|Output],Count+1);
false -> remove_unique([H2|T],[H1|Output],0)
end.
Test
7> test:remove_unique([1,2,3,3,3,5,5,6,7,7]).
[7,7,5,5,3,3,3]
8> test:remove_unique([1,2,3,3,3,5,5,6,7,8]).
[5,5,3,3,3]
I am doing somthing horrible but I don't know how to make it better.
I am forming all pairwise sums of the elements of a List called SomeList, but I don't want to see duplicates ( I guess I want "all possible pairwise sums" ):
sets:to_list(sets:from_list([A+B || A <- SomeList, B <- SomeList]))
SomeList does NOT contain duplicates.
This works, but is horribly inefficient, because the original list before the set conversion is GIGANTIC.
Is there a better way to do this?
You could simply use lists:usort/1
lists:usort([X+Y || X <- L, Y <- L]).
if the chance to have duplicates is very high, then you can generate the sum using 2 loops and store the sum in an ets set (or using map, I didn't check the performance of both).
7> Inloop = fun Inloop(_,[],_) -> ok; Inloop(Store,[H|T],X) -> ets:insert(Store,{X+H}), Inloop(Store,T,X) end.
#Fun<erl_eval.42.54118792>
8> Outloop = fun Outloop(Store,[],_) -> ok; Outloop(Store,[H|T],List) -> Inloop(Store,List,H), Outloop(Store,T,List) end.
#Fun<erl_eval.42.54118792>
9> Makesum = fun(L) -> S = ets:new(temp,[set]), Outloop(S,L,L), R =ets:foldl(fun({X},Acc) -> [X|Acc] end,[],S), ets:delete(S), R end.
#Fun<erl_eval.6.54118792>
10> Makesum(lists:seq(1,10)).
[15,13,8,11,20,14,16,12,7,3,10,9,19,18,4,17,6,2,5]
11> lists:sort(Makesum(lists:seq(1,10))).
[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
12>
This module will allow you to compare times of execution when using list comprehension, sets or ets. You can of course add additional functions to this comparison:
-module(pairwise).
-export([start/2]).
start(Type, X) ->
L = lists:seq(1, X),
timer:tc(fun do/2, [Type, L]).
do(compr, L) ->
sets:to_list(sets:from_list([A+B || A <- L, B <- L]));
do(set, L) ->
F = fun(Sum, Set) -> sets:add_element(Sum, Set) end,
R = fun(Set) -> sets:to_list(Set) end,
do(L, L, sets:new(), {F, R});
do(ets, L) ->
F = fun(Sum, Tab) -> ets:insert(Tab, {Sum}), Tab end,
R = fun(Tab) ->
Fun = fun({X}, Acc) -> [X|Acc] end,
Res = ets:foldl(Fun, [], Tab),
ets:delete(Tab),
Res
end,
do(L, L, ets:new(?MODULE, []), {F, R}).
do([A|AT], [B|BT], S, {F, _} = Funs) -> do([A|AT], BT, F(A+B, S), Funs);
do([_AT], [], S, {_, R}) -> R(S);
do([_A|AT], [], S, Funs) -> do(AT, AT, S, Funs).
Results:
36> {_, Res1} = pairwise:start(compr, 20).
{282,
[16,32,3,19,35,6,22,38,9,25,12,28,15,31,2,18,34,5,21,37,8,
24,40,11,27,14,30|...]}
37> {_, Res2} = pairwise:start(set, 20).
{155,
[16,32,3,19,35,6,22,38,9,25,12,28,15,31,2,18,34,5,21,37,8,
24,40,11,27,14,30|...]}
38> {_, Res3} = pairwise:start(ets, 20).
{96,
[15,25,13,8,21,24,40,11,26,20,14,28,23,16,12,39,34,36,7,32,
35,3,33,10,9,19,18|...]}
39> R1=lists:usort(Res1), R2=lists:usort(Res2), R3=lists:usort(Res3).
[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,
24,25,26,27,28,29,30|...]
40> R1 = R2 = R3.
[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,
24,25,26,27,28,29,30|...]
The last line is to compare that all functions return the same result but sorted differently.
First number in each resulted tuple is the time of execution as returned from timer:tc(fun do/2, [Type, L]).. In this example it's 282 for list comprehension, 155 for sets and 96 for ets.
An effective way is to use foldl instead of lists comprehension, because in this case you nedd a state on each step
sets:to_list(
lists:foldl(fun(A, S1) ->
lists:foldl(fun(B, S2) ->
sets:add_element(A+B, S2)
end, S1, SomeListA)
end, sets:new(), SomeListB)).
This solution keeps it relatively fast and makes use of as much pre-written library code as possible.
Note that I use lists:zip/2 here rather than numeric +, only to illustrate that this approach works for any kind of non-repeating permutation of a unique list. You may only care about arithmetic, but if you want more, this can do it.
-export([permute_unique/1]).
permute_unique([]) ->
[];
permute_unique([A|Ab]) ->
lists:zip(lists:duplicate(length(Ab)+1, A), [A|Ab])
++
permute_unique(Ab).
%to sum integers, replace the lists:zip... line with
% [B+C || {B,C} <- lists:zip(lists:duplicate(length(Ab)+1, A), [A|Ab])]
%to perform normal arithmetic and yield a numeric value for each element
I am not sure what you consider gigantic - you will end up with N*(N+1)/2 total elements in the permuted list for a unique list of N original elements, so this gets big really fast.
I did some basic performance testing of this, using an Intel (Haswell) Core i7 # 4GHz with 32GB of memory, running Erlang/OTP 17 64-bit.
5001 elements in the list took between 2 and 5 seconds according to timer:tc/1.
10001 elements in the list took between 15 and 17 seconds, and required about 9GB of memory. This generates a list of 50,015,001 elements.
15001 elements in the list took between 21 and 25 seconds, and required about 19GB of memory.
20001 elements in the list took 49 seconds in one run, and peaked at about 30GB of memory, with about 200 million elements in the result. That is the limit of what I can test.
How can I convert tuple from MongoDB
{'_id',<<"vasya">>,password,<<"12ghd">>,age,undefined}
to proplist
[{'_id',<<"vasya">>},{password,<<"12ghd">>},{age,undefined}]
Assuming you want to basically combine the two consecutive elements of the tuple together, this isn't too hard. You can use element\2 to pull elements from tuples. And tuple_size\1 to get the size of the tuple. Here's a couple of ways to handle this:
1> Tup = {'_id',<<"vasya">>,password,<<"12ghd">>,age,undefined}.
{'_id',<<"vasya">>,password,<<"12ghd">>,age,undefined}
2> Size = tuple_size(Tup).
6
You can use a list comprehension for this:
3> [{element(X, Tup), element(X+1, Tup)} || X <- lists:seq(1, Size, 2)].
[{'_id',<<"vasya">>},{password,<<"12ghd">>},{age,undefined}]
Or you can zip it:
4> lists:zip([element(X, Tup) || X <- lists:seq(1, Size, 2)], [element(X, Tup) || X <- lists:seq(2, Size, 2)]).
[{'_id',<<"vasya">>},{password,<<"12ghd">>},{age,undefined}]
You can clean up that zip by making a function to handle pulling elements out.
slice(Tuple, Start, Stop, Step) ->
[element(N, Tuple) || N <- lists:seq(Start, Stop, Step)].
Then calling this function:
5> lists:zip(slice(Tup, 1, Size, 2), Slice(Tup, 2, Size, 2)).
[{'_id',<<"vasya">>},{password,<<"12ghd">>},{age,undefined}]
You can use bson:fields/1 (https://github.com/mongodb/bson-erlang/blob/master/src/bson.erl#L52). bson is the dependency of mongodb erlang driver
Alternatively you could write a simple function to do it:
mongo_to_proplist(MongoTuple) ->
mongo_to_tuple(MongoTuple, 1, tuple_size(MongoTuple)).
mongo_to_proplist(Mt, I, Size) when I =:= Size -> [];
mongo_to_proplist(Mt, I, Size) ->
Key = element(I, Mt),
Val = element(I+1, Mt),
[{Key,Val}|mongo_to_proplist(Mt, I+2, Size)].
This is basically what the list comprehension versions are doing but unravelled into an explicit loop.
Not very efficient, so do not use it on large structures, but really simple:
to_proplist(Tuple) -> to_proplist1(tuple_to_list(Tuple), []).
to_proplist1([], Acc) -> Acc;
to_proplist1([Key, Value | T], Acc) -> to_proplist1(T, [{Key, Value} | Acc]).
If order should for some strange reason be important reverse the proplist in the base case of to_proplist1/2.
The following is a erlang function. I don't understand how lists:map function is used here.
Could someone please explain?
% perform M runs with N calls to F in each run.
% For each of the M runs, determine the average time per call.
% Return, the average and standard deviation of these M results.
time_it(F, N, M) ->
G = fun() -> F(), ok end,
NN = lists:seq(1, N),
MM = lists:seq(1, M),
T = lists:map(
fun(_) ->
T0 = now(), % start timer
[ G() || _ <- NN ], % make N calls to F
1.0e-6*timer:now_diff(now(), T0)/N % average time per call
end,
MM
),
{ avg(T), std(T) }.
Thanks.
also, I don't know the proper syntax when using this function. For example, I have a dummy() function take 1 parameter. I get an error while trying to time the dummy function.
moduleName:time_it(moduleName:dummy/1, 10, 100).
the above would evaluate to illegal expression.
Actually, now with the correct syntax, the function can be invoked correctly with:
moduleName:time_it(fun moduleName:dummy/1, 10, 100).
However, it will throw a exception saying invoking dummy function without passing any parameter. I think this line is the villain, [ G() || _ <- NN ], I have no idea how to fix it.
map is used here to execute the function
T0 = now(), % start timer
[ G() || _ <- NN ], % make N calls to F
1.0e-6*timer:now_diff(now(), T0)/N % average time per call
for each element of MM. map will return a new list of the same size, where each element of the new list is the result of applying the above function to the corresponding element of MM.
You can invoke time_it like:
moduleName:time_it(fun moduleName:dummy/1, 10, 100).
The purpose of lists:map in the time_it function is just to run the inner function M times. When you see this pattern:
L = lists:seq(1,M),
lists:map(fun(_)-> Foo() end, L)
It just means call Foo() again and again M times, and return the results of each call in a list. It actually makes a list of integers [1,2,3,...N] and then calls Foo() once for each member of the list.
The author of time_it does this same trick again, because time_it needs to call the function you give it N*M times. So inside the outer loop that runs M times they use a different technique to run the inner loop N times:
L = lists:seq(1,N),
[Foo() || _ <- L]
This has exactly the same result as the code above, but this time Foo is called N times.
The reason you are having trouble using time_it with your dummy function is that time_it takes a function with 0 parameters, not 1. So you need to make a dummy function and call it like this:
dummy() ->
%% do something here you want to measure
ok.
measure_dummy() ->
time_it(fun someModule:dummy/0, 10, 100).
If you have a function moduleName:dummy/1 you can do one of the following
If you can edit time_it/3, then make it call F(constant_parameter) instead of F(). I assume this is the case.
Otherwise, call M1:time_it(fun() -> M2:dummy(constant_parameter) end, N, M).
dummy will not be called directly, but only by F inside time_it.
results(N, F) when N >= 0 -> results(N, F, []).
results(0, _, Acc) -> lists:reverse(Acc);
results(N, F, Acc) -> results(N-1, F, [F() | Acc]).
repeat(0, F) -> ok;
repeat(N, F) when N > 0 ->
F(),
repeat(N-1, F).
With these:
T = results(M, fun () ->
T0 = now(),
repeat(N, G),
1.0e-6 * timer:now_diff(now(), T0)/N
end)
Make sense, now?