Related
I am writing a program that solves producers-consumers problem using Erlang multiprocessing with one process responsible for handling buffer to which I produce/consume and many producers and many consumers processes. To simplify I assume producer/consumer does not know that his operation has failed (that it is impossible to produce or consume because of buffer constraints), but the server is prepared to do this.
My code is:
Server code
server(Buffer, Capacity, CountPid) ->
receive
%% PRODUCER
{Pid, produce, InputList} ->
NumberProduce = lists:flatlength(InputList),
case canProduce(Buffer, NumberProduce, Capacity) of
true ->
NewBuffer = append(InputList, Buffer),
CountPid ! lists:flatlength(InputList),
Pid ! ok,
server(NewBuffer,Capacity, CountPid);
false ->
Pid ! tryagain,
server(Buffer, Capacity, CountPid)
end;
%% CONSUMER
{Pid, consume, Number} ->
case canConsume(Buffer, Number) of
true ->
Data = lists:sublist(Buffer, Number),
NewBuffer = lists:subtract(Buffer, Data),
Pid ! {ok, Data},
server(NewBuffer, Capacity,CountPid);
false ->
Pid ! tryagain,
server(Buffer, Capacity, CountPid)
end
end.
Producer and consumer
producer(ServerPid) ->
X = rand:uniform(9),
ToProduce = [rand:uniform(500) || _ <- lists:seq(1, X)],
ServerPid ! {self(),produce,ToProduce},
producer(ServerPid).
consumer(ServerPid) ->
X = rand:uniform(9),
ServerPid ! {self(),consume,X},
consumer(ServerPid).
Starting and auxiliary functions (I enclose as I don't know where exactly my problem is)
spawnProducers(Number, ServerPid) ->
case Number of
0 -> io:format("Spawned producers");
N ->
spawn(zad2,producer,[ServerPid]),
spawnProducers(N - 1,ServerPid)
end.
spawnConsumers(Number, ServerPid) ->
case Number of
0 -> io:format("Spawned producers");
N ->
spawn(zad2,consumer,[ServerPid]),
spawnProducers(N - 1,ServerPid)
end.
start(ProdsNumber, ConsNumber) ->
CountPid = spawn(zad2, count, [0,0]),
ServerPid = spawn(zad2,server,[[],20, CountPid]),
spawnProducers(ProdsNumber, ServerPid),
spawnConsumers(ConsNumber, ServerPid).
canProduce(Buffer, Number, Capacity) ->
lists:flatlength(Buffer) + Number =< Capacity.
canConsume(Buffer, Number) ->
lists:flatlength(Buffer) >= Number.
append([H|T], Tail) ->
[H|append(T, Tail)];
append([], Tail) ->
Tail.
I am trying to count number of elements using such process, server sends message to it whenever elements are produced.
count(N, ThousandsCounter) ->
receive
X ->
if
N >= 1000 ->
io:format("Yeah! We have produced ~p elements!~n", [ThousandsCounter]),
count(0, ThousandsCounter + 1000);
true -> count(N + X, ThousandsCounter)
end
end.
I expect this program to work properly, which means: it produces elements, increase of produced elements depends on time like f(t) = kt, k-constant and the more processes I have the faster production is.
ACTUAL QUESTION
I launch program:
erl
c(zad2)
zad2:start(5,5)
How the program behaves:
The longer production lasts the less elements in the unit of time are being produced (e.g. in first second 10000, in next 5000, in 10th second 1000 etc.
The more processes I have, the slower production is, in start(10,10) I need to wait about a second for first thousand, whereas for start(2,2) 20000 appears almost immediately
start(100,100) made me restart my computer (I work on Ubuntu) as the whole CPU was used and there was no memory available for me to open terminal and terminate erlang machine
Why does my program not behave like I expect? Am I doing something wrong with Erlang programming or is this the matter of OS or anything else?
The producer/1 and consumer/1 functions as written above don't ever wait for anything - they just loop and loop, bombarding the server with messages. The server's message queue is filling up very quickly, and the Erlang VM will try to grow as much as it can, stealing all your memory, and the looping processes will steal all available CPU time on all cores.
I understand that I can set a seq_trace in erlang to the current process that is executing. But how can I set it on another process from the shell, or remote shell like dbg tracing?
You can enable sequential tracing on another process using dbg. For example, let's say we have a module x with an exported call/2 function:
call(Pid, Msg) ->
Pid ! {self(), Msg},
receive
{Pid, Reply} -> Reply
end.
This function implements a simple call-response. Let's also say we have a module y that has a looping receiver function:
loop() ->
receive
{Pid, Msg} ->
seq_trace:print({?MODULE, self(), Pid, Msg}),
Pid ! {self(), {Msg, os:timestamp()}};
_ -> ok
end,
?MODULE:loop().
This function expects a message of the form sent by x:call/2, and when it receives one it prints a message into the sequential trace, if enabled, and then sends the original message back to the caller augmented with a timestamp. It ignores all other messages.
We also need a function to collect the sequential trace. The recursive systracer/1 function below just collects seq_trace tuples into a list, and produces the list of seq_trace messages when asked:
systracer(Acc) ->
receive
{seq_trace,_,_,_}=S ->
systracer([S|Acc]);
{seq_trace,_,_}=S ->
systracer([S|Acc]);
{dump, Pid} ->
Pid ! lists:reverse(Acc),
systracer([]);
stop -> ok
end.
Let's assume our systracer/1 function is exported from module x as well.
Let's use our Erlang shell to set this all up. First, let's spawn y:loop/0 and x:systracer/1:
1> Y = spawn(y,loop,[]).
<0.36.0>
2> S = spawn(x,systracer,[[]]).
<0.38.0>
3> seq_trace:set_system_tracer(S).
false
After spawning x:systracer/1 we set the process as the seq_trace system tracer. Now we need to start dbg:
4> dbg:tracer(), dbg:p(all,call).
{ok,[{matched,nonode#nohost,28}]}
These dbg calls are pretty standard, but you can feel free to vary them as needed especially if you plan to use dbg tracing during your debug session as well.
In practice when you enable sequential tracing with dbg, you typically do so by keying on a particular argument to a function. This enables you to get a trace specific to a given function invocation without getting traces for all invocations of that function. Along these lines, we'll use dbg:tpl/3 to turn on sequential trace flags when x:call/2 is invoked with its second argument having the value of the atom trace. First, we use dbg:fun2ms/1 to create the appropriate match specification to enable the sequential tracing flags we want, then we'll apply the match spec with dbg:tpl/3:
5> Ms = dbg:fun2ms(fun([_,trace]) -> set_seq_token(send,true), set_seq_token('receive',true), set_seq_token(print,true) end).
[{['_',trace],
[],
[{set_seq_token,send,true},
{set_seq_token,'receive',true},
{set_seq_token,print,true}]}]
6> dbg:tpl(x,call,Ms).
{ok,[{matched,nonode#nohost,1},{saved,1}]}
Now we can call x:call/2 with the second argument trace to cause sequential tracing to occur. We make this call from a spawned process to avoid having shell I/O-related messages appearing in the resulting trace:
7> spawn(fun() -> x:call(Y, trace), x:call(Y, foo) end).
(<0.46.0>) call x:call(<0.36.0>,trace)
<0.46.0>
The first line of output comes from normal dbg tracing, since we specified dbg:p(all, call) earlier. To get the sequential trace results, we need to get a dump from our systrace/1 process:
8> S ! {dump, self()}.
{dump,<0.34.0>}
This sends all sequential trace collected so far to our shell process. We can use the shell flush() command to view them:
9> flush().
Shell got [{seq_trace,0,{send,{0,1},<0.47.0>,<0.36.0>,{<0.47.0>,trace}}},
{seq_trace,0,{'receive',{0,1},<0.47.0>,<0.36.0>,{<0.47.0>,trace}}},
{seq_trace,0,{print,{1,2},<0.36.0>,[],{y,<0.36.0>,<0.47.0>,trace}}},
{seq_trace,0,
{send,{1,3},
<0.36.0>,<0.47.0>,
{<0.36.0>,{trace,{1423,709096,206121}}}}},
{seq_trace,0,
{'receive',{1,3},
<0.36.0>,<0.47.0>,
{<0.36.0>,{trace,{1423,709096,206121}}}}},
{seq_trace,0,{send,{3,4},<0.47.0>,<0.36.0>,{<0.47.0>,foo}}},
{seq_trace,0,{'receive',{3,4},<0.47.0>,<0.36.0>,{<0.47.0>,foo}}},
{seq_trace,0,{print,{4,5},<0.36.0>,[],{y,<0.36.0>,<0.47.0>,foo}}},
{seq_trace,0,
{send,{4,6},
<0.36.0>,<0.47.0>,
{<0.36.0>,{foo,{1423,709096,206322}}}}},
{seq_trace,0,
{'receive',{4,6},
<0.36.0>,<0.47.0>,
{<0.36.0>,{foo,{1423,709096,206322}}}}}]
And sure enough, these are the sequential trace messages we expected to see. First, for the message containing the trace atom, we have the send from x:call/2 followed by the reception in y:loop/0 and the result of seq_trace:print/1, then the send from y:loop/0 back to the caller of x:call/2. Then, since x:call(Y,foo) is called in the same process, which means all the sequential tracing flags are still enabled, the first set of sequential trace messages is followed by a similar set for the x:call(Y,foo) invocation.
If we just call x:call(Y,foo) we can see we get no sequential trace messages:
10> spawn(fun() -> x:call(Y, foo) end).
<0.55.0>
11> S ! {dump, self()}.
{dump,<0.34.0>}
12> flush().
Shell got []
This is because our match spec enables sequential tracing only when the second argument to x:call/2 is the atom trace.
For more information, see the seq_trace and dbg man pages, and also read the match specification chapter of the Erlang Run-Time System Application (ERTS)
User's Guide.
I am learning Erlang from a Ruby background and having some difficulty grasping the thought process. The problem I am trying to solve is the following:
I need to make the same request to an api, each time I receive a unique ID in the response which I need to pass into the next request until there is not ID returned. From each response I need to extract certain data and use it for other things as well.
First get the iterator:
ShardIteratorResponse = kinetic:get_shard_iterator(GetShardIteratorPayload).
{ok,[{<<"ShardIterator">>,
<<"AAAAAAAAAAGU+v0fDvpmu/02z5Q5OJZhPo/tU7fjftFF/H9M7J9niRJB8MIZiB9E1ntZGL90dIj3TW6MUWMUX67NEj4GO89D"...>>}]}
Parse out the shard_iterator..
{_, [{_, ShardIterator}]} = ShardIteratorResponse.
Make the request to kinesis for the streams records...
GetRecordsPayload = [{<<"ShardIterator">>, <<ShardIterator/binary>>}].
[{<<"ShardIterator">>,
<<"AAAAAAAAAAGU+v0fDvpmu/02z5Q5OJZhPo/tU7fjftFF/H9M7J9niRJB8MIZiB9E1ntZGL90dIj3TW6MUWMUX67NEj4GO89DETABlwVV"...>>}]
14> RecordsResponse = kinetic:get_records(GetRecordsPayload).
{ok,[{<<"NextShardIterator">>,
<<"AAAAAAAAAAFy3dnTJYkWr3gq0CGo3hkj1t47ccUS10f5nADQXWkBZaJvVgTMcY+nZ9p4AZCdUYVmr3dmygWjcMdugHLQEg6x"...>>},
{<<"Records">>,
[{[{<<"Data">>,<<"Zmlyc3QgcmVjb3JkISEh">>},
{<<"PartitionKey">>,<<"BlanePartitionKey">>},
{<<"SequenceNumber">>,
<<"49545722516689138064543799042897648239478878787235479554">>}]}]}]}
What I am struggling with is how do I write a loop that keeps hitting the kinesis endpoint for that stream until there are no more shard iterators, aka I want all records. Since I can't re-assign the variables as I would in Ruby.
WARNING: My code might be bugged but it's "close". I've never ran it and don't see how last iterator can look like.
I see you are trying to do your job entirely in shell. It's possible but hard. You can use named function and recursion (since release 17.0 it's easier), for example:
F = fun (ShardIteratorPayload) ->
{_, [{_, ShardIterator}]} = kinetic:get_shard_iterator(ShardIteratorPayload),
FunLoop =
fun Loop(<<>>, Accumulator) -> % no clue how last iterator can look like
lists:reverse(Accumulator);
Loop(ShardIterator, Accumulator) ->
{ok, [{_, NextShardIterator}, {<<"Records">>, Records}]} =
kinetic:get_records([{<<"ShardIterator">>, <<ShardIterator/binary>>}]),
Loop(NextShardIterator, [Records | Accumulator])
end,
FunLoop(ShardIterator, [])
end.
AllRecords = F(GetShardIteratorPayload).
But it's too complicated to type in shell...
It's much easier to code it in modules.
A common pattern in erlang is to spawn another process or processes to fetch your data. To keep it simple you can spawn another process by calling spawn or spawn_link but don't bother with links now and use just spawn/3.
Let's compile simple consumer module:
-module(kinetic_simple_consumer).
-export([start/1]).
start(GetShardIteratorPayload) ->
Pid = spawn(kinetic_simple_fetcher, start, [self(), GetShardIteratorPayload]),
consumer_loop(Pid).
consumer_loop(FetcherPid) ->
receive
{FetcherPid, finished} ->
ok;
{FetcherPid, {records, Records}} ->
consume(Records),
consumer_loop(FetcherPid);
UnexpectedMsg ->
io:format("DROPPING:~n~p~n", [UnexpectedMsg]),
consumer_loop(FetcherPid)
end.
consume(Records) ->
io:format("RECEIVED:~n~p~n",[Records]).
And fetcher:
-module(kinetic_simple_fetcher).
-export([start/2]).
start(ConsumerPid, GetShardIteratorPayload) ->
{ok, [ShardIterator]} = kinetic:get_shard_iterator(GetShardIteratorPayload),
fetcher_loop(ConsumerPid, ShardIterator).
fetcher_loop(ConsumerPid, {_, <<>>}) -> % no clue how last iterator can look like
ConsumerPid ! {self(), finished};
fetcher_loop(ConsumerPid, ShardIterator) ->
{ok, [NextShardIterator, {<<"Records">>, Records}]} =
kinetic:get_records(shard_iterator(ShardIterator)),
ConsumerPid ! {self(), {records, Records}},
fetcher_loop(ConsumerPid, NextShardIterator).
shard_iterator({_, ShardIterator}) ->
[{<<"ShardIterator">>, <<ShardIterator/binary>>}].
As you can see both processes can do their job concurrently.
Try from your shell:
kinetic_simple_consumer:start(GetShardIteratorPayload).
Now your see that your shell process turns to consumer and you will have your shell back after fetcher will send {ItsPid, finished}.
Next time instead of
kinetic_simple_consumer:start(GetShardIteratorPayload).
run:
spawn(kinetic_simple_consumer, start, [GetShardIteratorPayload]).
You should play with spawning processes - it's erlang main strength.
In Erlang, you can write loop using tail recursive functions. I don't know the kinetic API, so for simplicity, I just assume, that kinetic:next_iterator/1 return {ok, NextIterator} or {error, Reason} when there are no more shards.
loop({error, Reason}) ->
ok;
loop({ok, Iterator}) ->
do_something_with(Iterator),
Result = kinetic:next_iterator(Iterator),
loop(Result).
You are replacing loop with iteration. First clause deals with case, where there are no more shards left (always start recursion with the end condition). Second clause deals with case, where we got some iterator, we do something with it and call next.
The recursive call is last instruction in the function body, which is called tail recursion. Erlang optimizes such calls - they don't use call stack, so they can run infinitely in constant memory (you will not get anything like "Stack level too deep")
I am developing a simple framework in Erlang to handle 2-player turn-based games. The code is the following:
-module(game).
-export([start_server/0,generate_server/0,add_player/0,remove_player/0]).
generate_server() ->
Table_num = 0,
Player_num = 0,
io:format("Server generated...~n", []),
io:format("The current number of tables is ~w~n", [Table_num]),
io:format("The current number of players is ~w~n", [Player_num]),
receive
login ->
io:format("A new player has connected!~n", []),
New = Player_num + 1,
io:format("The current number of players is ~w~n", [New]);
logout ->
io:format("You have beeen succesfully disconnected~n", [])
end.
start_server() ->
io:format("Welcome player!~nInitializing game...~n", []),
io:format("Generating server...~n", []),
register(server,spawn(game, generate_server, [])).
add_player() ->
server ! login.
remove_player() ->
server ! logout.
There are two main problems when I run this code:
When I execute add_player() and then remove_player(), this second function crashes with an exception
If I launch the program on one terminal window and then execute add_player() on a second terminal windows, I get an error. What should I do to be able to run it on more than one terminal window?
Any help would be highly appreciated.
1/
There is no loop in your server. When you start it, after some print, it waits on the receive statement.
When it receives the login message, it executes the operations, and then the server code is finished; the process die and is unregistered. All variables disappear and the VM clean up the memory...
So, later on, any process sending a message to the server will crash because you are using a name which is no more registered.
To make it works you should keep a list of connected players (in list for example) and recall the server loop with this list as parameter.
generate_server(Tlist,Plist) ->
io:format("The current number of tables is ~w~n", [length(Tlist)]),
io:format("The current number of players is ~w~n", [length(Plist)]),
receive
{login,Name} ->
io:format("A new player ~p has connected!~n", [Name]),
generate_server(Tlist,[Name|Plist]);
{logout,Name} ->
io:format("You have beeen succesfully disconnected~n", []),
generate_server(Tlist,lists:delete(Name,Plist))
end.
and the call to generate_server is done by
register(server,spawn(game, generate_server, [[],[]]))
2/
in order to use erlang messages between 2 different nodes, you need to:
share the same erlang coockie
discover the nodes (using netadm for example)
get the server pid or use golbal registered name
see example at http://learnyousomeerlang.com/distribunomicon#alone-in-the-dark
You missed out a loop body for the server. Your program crashes because the server receives only one message and exits. Consider another version of the server below:
generate_server() ->
Table_num = 0,
Player_num = 0,
io:format("Server generated...~n", []),
io:format("The current number of tables is ~w~n", [Table_num]),
io:format("The current number of players is ~w~n", [Player_num]),
loop([]).
loop(Players)->
receive
{From,{login,PlayerId}} ->
io:format("A new player has connected!~n", []),
io:format("The current number of players is ~w~n", [New]),
NewPlayers = case lists:member(PlayerId,Players) of
true ->
From ! {login_failed,exists},
Players;
false ->
From ! {login_success,true},
[PlayerId|Players]
end,
loop(NewPlayers);
{From,{logout,PlayerId}} ->
NewPlayers = case lists:member(PlayerId,Players) of
true ->
From ! {logout,ok},
Players -- [PlayerId];
false ->
From ! {logout,failed},
Players
end, loop(NewPlayers);
_ -> loop(Players)
end.
There; that looks much better.
I am implementing a car park with 2 entry gates and 1 by which you can leave the park. For me, everything looks fine but I am getting errors like
Error in process <0.84.0> with exit value: {badarg,[{parking,car,2},{random,uniform,0}]}
My code is:
-module (parking2).
-export ([start/3]).
-export ([car/2, parkingLoop/1]).
carsInit(0, _Iterations) ->
ok;
carsInit(Number, Iterations) ->
spawn(parking, car, [Number, Iterations]),
carsInit(Number - 1, Iterations).
car(_ID, 0) ->
ok;
car(ID, Iterations) ->
Gate = random:uniform(2),
parking ! {enter, self()},
receive
error ->
io:format("Car ~B ncanot enter - there is no free place.~n", [ID]),
Time = random:uniform(1000),
timer:sleep(Time),
car(ID, Iterations);
ok ->
io:format("Car ~B entered through the ~B gate~n", [ID, Gate])
end,
StopTime = random:uniform(500) + 500,
timer:sleep(StopTime),
parking ! {leave, self(), ID},
FreerideTime = random:uniform(1000) + 500,
timer:sleep(FreerideTime),
car(ID, Iterations - 1).
parkingInit(Slots) ->
spawn(parking, parkingLoop, [Slots]).
parkingLoop(Slots) ->
receive
{enter, Pid} ->
if Slots =:= 0 ->
Pid ! error
end,
Pid ! ok,
parkingLoop(Slots - 1);
{leave, Pid, ID} ->
io:format("Car ~B left the car park.", [ID]),
parkingLoop(Slots + 1);
stop ->
ok
end.
start(Cars, Slots, Iterations) ->
parkingInit(Slots),
carsInit(Cars, Iterations).
May anybody help me ? I learn Erlang for a couple of days and have no idea, what's wrong here.
Thanks in advance,
Radek
The example you posted uses the wrong module in the spawn/3 call:
spawn(parking, parkingLoop, [Slots]).
It should work better (or at least give a more up to date error) if you change this to:
spawn(?MODULE, parkingLoop, [Slots]).
(Always use ?MODULE, which is a macro that evaluates to the current module name, when doing such things since it will avoid a lot of mistakes using the wrong module than intended).
The bug comes from not registering the parkingLoop process. You're trying to send a message to it using parking ! ... but no process is named parking. Change line 33 to:
register(parking, spawn(parking2, parkingLoop, [Slots])).
(Even here you can use the ?MODULE macro to avoid problems in the future: ?MODULE ! ... and register(?MODULE, ...) since you only have one process with this name)
Also, your if statement on line 38 misses a fall-through clause. Make it look like this to handle the case where Slots is not equal to zero:
if
Slots =:= 0 ->Pid ! error;
true -> ok
end,
(The ok expression will have no effect since the return value of the if statement is not used)