I tried this code snippet:
print_next(Current) ->
case mnesia:dirty_next(muppet, Current) of
'$end_of_table' ->
io:format("~n", []),
ok;
Next ->
[Muppet] = mnesia:dirty_read({muppet, Next}),
io:format("~p~n", [Muppet]),
print_next(Next),
ok
end.
print() ->
case mnesia:dirty_first(muppet) of
'$end_of_table' ->
ok;
First ->
[Muppet] = mnesia:dirty_read({muppet, First}),
io:format("~p~n", [Muppet]),
print_next(First),
ok
end.
But it is so long. Also I can use dirty_all_keys and then iterate through key list, but I want to know if there is a better way to print out Mnesia table contents.
If you just want a quick and dirty way to print the contents of a Mnesia table in the shell, and if your table is not of type disc_only_copies, then you can take advantage of the fact that Mnesia stores its data in ETS tables and run:
ets:tab2list(my_table).
or, if you think the shell truncates the data too much:
rp(ets:tab2list(my_table)).
Not recommended for "real" code, of course.
For a simple and quick look at your table contents you can use select function of mnesia with catch-all Match Specification as follows:
CatchAll = [{'_',[],['$_']}].
mnesia:dirty_select(TableName, CatchAll).
and also you can run it inside a transaction context:
CatchAll = [{'_',[],['$_']}].
SelectFun = fun() -> mnesia:select(TableName, CatchAll) end.
mnesia:transaction(SelectFun).
however be careful if you are in a production environment with a big data.
Well, if the intent is to see the contents of your table, there is the application called tv, which can view both ETS and mnesia tables.
If you wish to see all the table contents on your terminal, then try something like this:
traverse_table_and_show(Table_name)->
Iterator = fun(Rec,_)->
io:format("~p~n",[Rec]),
[]
end,
case mnesia:is_transaction() of
true -> mnesia:foldl(Iterator,[],Table_name);
false ->
Exec = fun({Fun,Tab}) -> mnesia:foldl(Fun, [],Tab) end,
mnesia:activity(transaction,Exec,[{Iterator,Table_name}],mnesia_frag)
end.
Then if your table is called muppet, you use the function as follows:
traverse_table_and_show(muppet).
Advantages of this:
If its executed within a transaction , it will have no problems of nested transactions. It is less work because its done within one mnesia transaction through mnesia iterator functionality as compared to your implementation of get_next_key -> do_read_with_key -> then read the record (these are many operations). With this, mnesia will automatically tell that it has covered all the records in your entire table. Also, if the table is fragmented, your functionality will only display records in the first fragment. This will iterate through all the fragments the belong to that table.
In this iteration mnesia method, i do nothing with the Accumulator variable which should go along with the Iterator fun and thats why you see the underscore for the second variable.
Details of this iteration can be found here: http://www.erlang.org/doc/man/mnesia.html#foldl-3
As Muzaaya told, you can you use tv (table visualizer tool) to view both mnesia and ets tables.
Alternatively, you can use the following code to get mnesia table data - Print on terminal or in case you want to store the result in a file :
select_all() ->
mnesia:transaction(
fun() ->
P=qlc:e(qlc:q([E || E <- mnesia:table(tableName)])), %query to select all data from table named 'tableName'
io:format(" ~p ~n ", [P]), % Prints table data on terminal
to_file("fileName.txt",P) % to_file method writes the data to file
end ).
to_file(File, L) ->
mnesia:transaction(
fun() ->
{ok, S} = file:open(File, write),
lists:foreach(fun(X) -> io:format(S, "~p.~n" ,[X]) end, L),
file:close(S)
end).
Related
I ma new in erlang and I need to do some operations for all records I get from mnesia database.
Result = mnesia:dirty_read(mydatabase, {key1, key2}),
case Result of
[] ->
?DEBUG("No such record found", []);
[#mydatabase{key3 = Key3}] ->
%% some operations
end
How can I add a loop to my code that execute some operations for all records?
I am not even sure if the code above does it or not?
You could use mnesia:foldl/3 for that. It iterates over all records in a table, passing along an "accumulator" value.
It doesn't have an explicit "dirty" counterpart, so if you want to run it as a dirty operation you need to use mnesia:activity/2. (Or you could just use it inside a call to mnesia:transaction.)
In this example, I don't actually do anything with the "accumulator", leaving as ignored_acc throughout.
mnesia:activity(sync_dirty,
fun() ->
mnesia:foldl(
fun(#mydatabase{}, Acc) ->
%% do something with the record here
Acc
end,
ignored_acc,
my_table)
end)
I think you can try all_keys(Tab)
all_keys(Tab) -> KeyList | transaction abort
This function returns a list of all keys in the table named Tab. The
semantics of this function is context sensitive. See mnesia:activity/4
for more information. In transaction context it acquires a read lock
on the entire table.
So I have made 2 databases:
Db1 that contains: [{james,london}]
Db2 that contains: [{james,london},{fredrik,berlin},{fred,berlin}]
I have a match function that looks like this:
match(Element, Db) -> proplists:lookup_all(Element, Db).
When I do: match(berlin, Db2) I get: [ ]
What I am trying to get is a way to input the value and get back the keys in this way: [fredrik,fred]
Regarding to documentation proplists:lookup_all works other way:
Returns the list of all entries associated with Key in List.
So, you can lookup only by keys:
(kilter#127.0.0.1)1> Db = [{james,london},{fredrik,berlin},{fred,berlin}].
[{james,london},{fredrik,berlin},{fred,berlin}]
(kilter#127.0.0.1)2> proplists:lookup_all(berlin, Db).
[]
(kilter#127.0.0.1)3> proplists:lookup_all(fredrik, Db).
[{fredrik,berlin}]
You can use lists:filter and lists:map instead:
(kilter#127.0.0.1)7> lists:filter(fun ({K, V}) -> V =:= berlin end, Db).
[{fredrik,berlin},{fred,berlin}]
(kilter#127.0.0.1)8> lists:map(fun ({K,V}) -> K end, lists:filter(fun ({K, V}) -> V =:= berlin end, Db)).
[fredrik,fred]
So, finally
match(Element, Db) -> lists:map(
fun ({K,V}) -> K end,
lists:filter(fun ({K, V}) -> V =:= Element end, Db)
).
proplists:lookup_all/2 takes as a first argument a key; in your example, berlin is a value and it's not a key therefore an empty list is returned.
Naturally, you can use recursion and find all the elements (meaning that you will use it like an ordinary list and not a proplist).
Another solution is to change the encoding scheme:
[{london,james},{berlin,fredrik},{berlin,fred}]
and then use proplists:lookup_all/2
The correct way to encode it depends on the way you will access the data (what kind of "queries" you will perform most); but unless you manipulate large amounts of data (in which case you might want to use some other datastructure) it isn't really worth analyzing.
I am learning Erlang and trying to figure out how I can, and should, save state inside a process.
For example, I am trying to write a program that given a list of numbers in a file, tells me whether a number appears in that file. My approach is to uses two processes
cache which reads the content of the file into a set, then waits for numbers to check, and then replies whether they appear in the set.
is_member_loop(Data_file) ->
Numbers = read_numbers(Data_file),
receive
{From, Number} ->
From ! {self(), lists:member(Number, Numbers)},
is_member_loop(Data_file)
end.
client which sends numbers to cache and waits for the true or false response.
check_number(Number) ->
NumbersPid ! {self(), Number},
receive
{NumbersPid, Is_member} ->
Is_member
end.
This approach is obviously naive since the file is read for every request. However, I am quite new at Erlang and it is unclear to me what would be the preferred way of keeping state between different requests.
Should I be using the process dictionary? Is there a different mechanism I am not aware of for that sort of process state?
Update
The most obvious solution, as suggested by user601836, is to pass the set of numbers as a param to is_member_loop instead of the filename. It seems to be a common idiom in Erlang and there is a good example in the fantastic online book Learn you some Erlang.
I think, however, that the question still holds for more complex state that I'd want to preserve in my process.
Simple solution, you can pass to your function is_member_loop(Data_file) the list of numbers rather then the file name.
The best solution when you deal with a state consists in using a gen_server. To learn more you should take a look at records and gen_server behaviour (this may also be useful).
In practice:
1) start with a module (yourmodule.erl) based on gen_server behaviour
2) read your file in the init function of the gen_server and pass it as state field:
init([]) ->
Numbers = read_numbers(Data_file),
{ok, #state{numbers=Numbers}}.
3) write a function which will be used to trigger a call to the gen_server
check_number(Number) ->
gen_server:call(?MODULE, {check_number, Number}).
4) write the code in order to handle messages triggered from your function
handle_call({check_number, Number}, _From, #state{numbers=Numbers} = State) ->
Reply = lists:member(Number, Numbers)},
{reply, Reply, State};
handle_call(_Request, _From, State) ->
Reply = ok,
{reply, Reply, State}.
5) export from yourmodule.erl function check_number
-export([check_number/1]).
Two things to be explained about point 4:
a) we extract values inside the record State using pattern matching
b) As you may see I left the generic handle call, otherwise your gen_server will fail due to wrong pattern matching whenever a message different from {check_number, Number} is received
Note: if you are new to erlang, don't use process dictionary
Not sure how idiomatic this is, since I'm not exactly an Erlang pro yet, but I'd handle this by using ETS. Basically,
read_numbers_to_ets(DataFile) ->
Table = ets:new(numbers, [ordered_set]),
insert_numbers(Table, DataFile),
Table.
insert_numbers(Table, DataFile) ->
case read_next_number(DataFile) of
eof -> ok;
Num -> ets:insert(numbers, {Num})
end.
you could then define your is_member as
is_member(TableId, Number) ->
case ets:match(TableId, {Number}) of
[] -> false; %% no match from ets
[[]] -> true %% ets found the number you're looking for in that table
end.
Instead of taking a Data_file, your is_member_loop would take the id of the table to do a lookup on.
I have two records:
-record(foo, {timestamp, name}).
-record(bar, {timestamp, name}).
And I would like to execute a Mnesia query that mimics the following SQL query
SELECT f.* FROM foo f WHERE f.timestamp NOT IN ( SELECT b.timestamp FROM boo b)
What would be an efficient Mnesia equivalent?
Good Question ! Now, i have thought of two ways. One where we use qlc and another where we use mnesia's own table iteration methods with accumulators. Here is the first option:
%% Here i use 'qlc', with a guard
%% which ensures that the timestamp
%% in the given 'foo record'
%% does NOT exist in table 'bar'
query()->
Is_not_in_bar = fun(Stamp)->
case mnesia:dirty_read({bar,Stamp}) of
[] -> true;
_ -> false
end
end,
Q = qlc:q([R || R <- mnesia:table(foo), Is_not_in_bar(R#foo.timestamp) == true])),
Fun = fun(QH)-> qlc:e(QH) end,
mnesia:activity(transaction,Fun,[Q],mnesia_frag).
Another option would be to iterate table foo while cross referencing each timestamp in the table bar. If its not found in bar then its added to the accumulated amount. look at this below
%% Here i iterate table 'foo'
%% for each record i find,
%% i try to cross reference
%% its timestamp in table 'bar'
%% If its not found, then i add that
%% 'foo record' into the Buffer
%% I accumulate this Buffer until
%% the entire table 'foo' has been
%% traversed
query_vsn2()->
Iterator = fun(#foo{timestamp = T} = Rec,Buffer)->
NewBuffer = case mnesia:dirty_read({bar,T}) of
[] -> [Rec|Buffer];
_ -> Buffer
end,
NewBuffer
end,
Find = fun(Loop)-> mnesia:foldl(Loop,[],foo) end,
mnesia:activity(transaction,Find,[Iterator],mnesia_frag).
I guess depending on the table size, the application and the user preference, each of these functions will have consequences. However, try both of them and see which one blends well into your app. The good thing is that this is entirely a read job, no writes, so i expect to be efficient enough. Success !
am josh in Uganda. i created a mnesia fragmented table (64 fragments), and managed to populate it upto 9948723 records. Each fragment was a disc_copies type, with two replicas.
Now, using qlc (query list comprehension), was too slow in searching for a record, and was returning inaccurate results.
I found out that this overhead is that qlc uses the select function of mnesia which traverses the entire table in order to match records. i tried something else below.
-define(ACCESS_MOD,mnesia_frag).
-define(DEFAULT_CONTEXT,transaction).
-define(NULL,'_').
-record(address,{tel,zip_code,email}).
-record(person,{name,sex,age,address = #address{}}).
match()-> Z = fun(Spec) -> mnesia:match_object(Spec) end,Z.
match_object(Pattern)->
Match = match(),
mnesia:activity(?DEFAULT_CONTEXT,Match,[Pattern],?ACCESS_MOD).
Trying this functionality gave me good results. But i found that i have to dynamically build patterns for every search that may be made in my stored procedures.
i decided to go through the havoc of doing this, so i wrote functions which will dynamically build wild patterns for my records depending on which parameter is to be searched.
%% This below gives me the default pattern for all searches ::= {person,'_','_','_'}
pattern(Record_name)->
N = length(my_record_info(Record_name)) + 1,
erlang:setelement(1,erlang:make_tuple(N,?NULL),Record_name).
%% this finds the position of the provided value and places it in that
%% position while keeping '_' in the other positions.
%% The caller function can use this function recursively until
%% it has built the full search pattern of interest
pattern({Field,Value},Pattern_sofar)->
N = position(Field,my_record_info(element(1,Pattern_sofar))),
case N of
-1 -> Pattern_sofar;
Int when Int >= 1 -> erlang:setelement(N + 1,Pattern_sofar,Value);
_ -> Pattern_sofar
end.
my_record_info(Record_name)->
case Record_name of
staff_dynamic -> record_info(fields,staff_dynamic);
person -> record_info(fields,person);
_ -> []
end.
%% These below,help locate the position of an element in a list
%% returned by "-record_info(fields,person)"
position(_,[]) -> -1;
position(Value,List)->
find(lists:member(Value,List),Value,List,1).
find(false,_,_,_) -> -1;
find(true,V,[V|_],N)-> N;
find(true,V,[_|X],N)->
find(V,X,N + 1).
find(V,[V|_],N)-> N;
find(V,[_|X],N) -> find(V,X,N + 1).
This was working very well though it was computationally intensive.
It could still work even after changing the record definition since at compile time, it gets the new record info
The problem is that when i initiate even 25 processes on a 3.0 GHz pentium 4 processor running WinXP, It hangs and takes a long time to return results.
If am to use qlc in these fragments, to get accurate results, i have to specify which fragment to search in like this.
find_person_by_tel(Tel)->
select(qlc:q([ X || X <- mnesia:table(Frag), (X#person.address)#address.tel == Tel])).
select(Q)->
case ?transact(fun() -> qlc:e(Q) end) of
{atomic,Val} -> Val;
{aborted,_} = Error -> report_mnesia_event(Error)
end.
Qlc was returning [], when i search for something yet when i use match_object/1 i get accurate results. I found that using match_expressions can help.
mnesia:table(Tab,Props).
where Props is a data structure that defines the match expression, the chunk size of return values e.t.c
I got a problem when i tried building match expressions dynamically.
Function mnesia:read/1 or mnesia:read/2 requires that you have the primary key
Now am asking myself, how can i efficiently use QLC to search for records in a large fragmented table? Please help.
I know that using tuple representation of records makes code hard to upgrade. This is why
i hate using mnesia:select/1, mnesia:match_object/1 and i want to stick to QLC. QLC is giving me wrong results in my queries from a mnesia table of 64 fragments even on the same node.
Has anyone ever used QLC to query a fragmented table?, please help
Do you invoke the qlc in the activity context?
tfn_match(Id) ->
Search = #person{address=#address{tel=Id, _ = '_'}, _ = '_'},
trans(fun() -> mnesia:match_object(Search) end).
tfn_qlc(Id) ->
Q = qlc:q([ X || X <- mnesia:table(person), (X#person.address)#address.tel == Id]),
trans(fun() -> qlc:e(Q) end).
trans(Fun) ->
try Res = mnesia:activity(transaction, Fun, mnesia_frag),
{atomic, Res}
catch exit:Error ->
{aborted, Error}
end.