excuse me, I'm quite not clear about these below descriptions at the erlang document:
erlang:memory() -> [{Type, Size}]
with Type: "total" means: "The total amount of memory currently allocated. This is the same as the sum of the memory size for processes and system."
memsup:get_system_memory_data() -> MemDataList
MemDataList = [{Tag, Size}]
with Tag: "total_memory" means: "The total amount of memory available to the Erlang emulator, allocated and free. May or may not be equal to the amount of memory configured in the system."
Does anyone here please make is more understable please ?
erlang:memory/0,1 shows the current state of Erlang VM (BEAM). memsup:get_system_memory_data/0 show the current state of OS.
1> erlang:memory(total) / (1 bsl 20).
12.918495178222656
2> memsup:get_system_memory_data().
[]
3>
=WARNING REPORT==== 13-Jan-2018::10:51:15 ===
OS_MON (memsup) called by <0.61.0>, not started
3> application:ensure_all_started(os_mon).
=PROGRESS REPORT==== 13-Jan-2018::10:51:41 ===
...
=INFO REPORT==== 13-Jan-2018::10:51:41 ===
alarm_handler: {set,{system_memory_high_watermark,[]}}
4> proplists:get_value(total_memory,memsup:get_system_memory_data()) / (1 bsl 30).
7.691337585449219
5> erlang:memory(total) / (1 bsl 20).
14.661575317382813
It means my BEAM was using 12.9MB memory after the start. For obtaining OS memory you have to start os_mon application. My OS has 7.69GB available which could be potentially used by BEAM. My BEAM was using 14.7MB after started all application necessary for os_mon.
Related
Actors send messages to one another. If the queues are limited, then what happens on write/send attempts to full queues? Blocking or dropping? If they are not limited, a memory crash is possible. How much is configurable?
Default mailboxes in Akka are not bounded, so will not prevent memory crash. You can however configure actors to use different mailboxes, among those there are both mailboxes that discard (pass to dead letters) messages when the max size is reached and those that block (I would not recommend to use those). You can find all mailbox implementations that comes with Akka in the docs here: https://doc.akka.io/docs/akka/current/typed/mailboxes.html#mailbox-implementations
You can test easily the behavior of the Erlang VM in this situation. In the shell:
F = fun F() -> receive done -> ok end end,
P = spawn(F),
G = fun G(Pid,Size,Wait) -> Pid ! lists:seq(1,Size), receive done -> ok after Wait -> G(Pid,Size,Wait) end end,
H = fun(Pid,Size,Wait) -> T = fun() -> G(Pid,Size,Wait) end, spawn(T) end,
D = fun D() -> io:format("~p~n~p~n",[erlang:time(),erlang:memory(processes_used)]), receive done -> ok after 10000 -> D() end end,
P1 = spawn(D).
P2 = H(P,100000,5).
You will see that you get a memory allocation exception, the VM writes a core dump and crashes.
I didn't check how to modify the limits, if you make the trial, you will see that it needs to reach a very high number of messages, using tens gigabytes of memory in the mailbox.
If you ever reach this situation, I don't think the first reaction is to increase the size, you should look first for
unread messages,
process bottleneck
application architecture
is Erlang adapted to your problem
...
actor queue in erlang not have limitation, this limited by memory size of VM, if memory size in VM is full VM crashed. for monitor or and management memory allocation and cpu load you can use os_mon in Erlang
you can test in erlang shell
F = fun() -> timer:sleep(60000),
{message_queue_len, InboxLen} = erlang:process_info(self(), message_queue_len),
io:format("Len ===> ~p", [InboxLen])
end.
PID = erlang:spawn(F).
[PID ! "hi" || _ <- lists:seq(1, 50000)].
if you increase number of message you can overflow memory
Default mailboxes in Akka are not bounded. But if you want to limit the max messages in mailboxes, you could build an Akka stream in the actor, then OverflowStrategy can be used on demand.
For example:
val source: Source[Message, SourceQueueWithComplete[Message]] =
Source.queue[Message](bufferSize = 8192,
overflowStrategy = OverflowStrategy.dropNew)
Can you give some examples where a process gets restarted by Erlang supervisor. If a process dies, it will restart. But how does a process die?
Thanks.
You can take as example what occurs in the Erlang shell, for example consider the sequence:
1> self().
<0.32.0>
2> A = 1.
1
3> self().
<0.32.0>
4> A = 2.
** exception error: no match of right hand side value 2
5> self().
<0.37.0>
1> The first command asks to the shell to prompt its own Pid: <0.32.0>.
2> Next a new command set the variable A to 1, it works, since A was unbound.
3> A new request to the shell shows that its Pid didn't change.
4> trying to match A with the integer 2 fails, it raise an exception. In fact, in the background, the shell process dies, and a supervisor restart it immediately.
5> It can be verified with a new request to get the shell Pid, now it is <0.37.0>.
6> when the shell died, it has lost every information, and it is restarted from scratch. But during initialization it can connect to some other processes who was in charge of keeping the history of the session, and all the bound variables. It can be verified by asking the value of A:
6> A.
1
7> or by asking the history
7> h().
1: self()
-> <0.32.0>
2: A = 1
-> 1
3: self()
-> <0.32.0>
4: A = 2
-> {'EXIT',{{badmatch,2},[{erl_eval,expr,3,[]}]}}
5: self()
-> <0.37.0>
6: A
Depending on the environment (hardware failure, loss of communication, bad parameters, bug...) an erlang process may die with an Error reason. If it is managed in a supervision tree (or your own monitoring) it can be restarted from scratch. It is the application responsibility to provide the means to all the processes to recover the appropriate state.
An erlang process may also die with the reason "normal", for example when a user close a session (in the shell you type q().), in this case, the supervisor will not restart it.
You will find many valuable information on the web:
design principle
erlang.org supervisor
learn you some erlang : run time errors
learn you some erlang : errors and processes
learn you some erlang : supervisors
First I have to mention that I run on a CentOS 7 tuned up to support 1 million connections. I tested with a simple C server and client and I connected 512000 clients. I could have connect more but I did not have enought RAM to spawn more linux client machines, since from a machine I can open 65536 connections; 8 machines * 64000 connections each = 512000.
I made a simple Erlang server to which I want to connect 1 million or half a million clients, using the same C client. The problem I'm having now is memory related. For each successfully gen_tcp:accept call I spawn a process. Around 50000 open connections costs me 3.7 GB RAM on server, meanwhile using the C server I could have open 512000 connections using 1.9 GB RAM. It is true that on the C server I did not created a process after accept to handle stuff, I just called accept again in while loop, but even so... guys on web did this erlang thing with less memory ( ejabberd riak )
I presume that the flags that I pass to the erlang VM should do the trick. From what I read in documentation and on the web this is what I have: erl +K true +Q 64200 +P 134217727 -env ERL_MAX_PORTS 40960000 -env ERTS_MAX_PORTS 40960000 +a 16 +hms 1024 +hmbs 1024
This is the server code, I open 1 listener that monitors port 5001 by calling start(1, 5001).
start(Num,LPort) ->
case gen_tcp:listen(LPort,[{reuseaddr, true},{backlog,9000000000}]) of
{ok, ListenSock} ->
start_servers(Num,ListenSock),
{ok, Port} = inet:port(ListenSock),
Port;
{error,Reason} ->
{error,Reason}
end.
start_servers(0,_) ->
ok;
start_servers(Num,LS) ->
spawn(?MODULE,server,[LS,0]),
start_servers(Num-1,LS).
server(LS, Nr) ->
io:format("before accept ~w~n",[Nr]),
case gen_tcp:accept(LS) of
{ok,S} ->
io:format("after accept ~w~n",[Nr]),
spawn(ex,server,[LS,Nr+1]),
proc_lib:hibernate(?MODULE, loop, [S]);
Other ->
io:format("accept returned ~w - goodbye!~n",[Other]),
ok
end.
loop(S) ->
ok = inet:setopts(S,[{active,once}]),
receive
{tcp,S, _Data} ->
Answer = 1, % Not implemented in this example
gen_tcp:send(S,Answer),
proc_lib:hibernate(?MODULE, loop, [S]);
{tcp_closed,S} ->
io:format("Socket ~w closed [~w]~n",[S,self()]),
ok
end.
Given this configuration your my beam consumed about 2.5 GB of memory just on start without even your module loaded.
However, if you reduce maximum number of processes to the reasonable value, like +P 60000 for 50 000 connections test, memory consumption drops rapidly.
With 60 000 processes limit VM only used 527MB of virtual memory on start.
I've tried to reproduce your test, but unfortunately I was only able to launch 30 000 netcat's on my system before running out of memory (because of client jobs). However I only observed increase of VM memory consumption up to 570MB.
So my suggestion is that your numbers come from high startup memory consumption and not great number of opened connections. Even then you actually should pay attention to the stats change along with increasing number of opened connections and not absolute values.
I finally used the following configuration for my benchmark:
erl +K true +Q 64200 +P 60000 -env ERL_MAX_PORTS 40960000 -env ERTS_MAX_PORTS 40960000 +a 16 +hms 1024 +hmbs 1024
So I've launched clients with the command
for i in `seq 1 50000`; do nc 127.0.0.1 5001 & done
Apart from tunes you already made you can adjust tcp buffers as well. By default they take OS default values, but you can pass {recbuf, Size}and {sndbuf, Size} to gen_tcp:listen. It may reduce memory footprints significantly.
I got this error report when running my Erlang application.
Crash dump was written to: erl_crash.dump
eheap_alloc: Cannot allocate 18446744071692551144 bytes of memory (of type "heap").
It's a simple program ran on a simple PC. How is it possible to get such numbers? It is trying to allocate 10^10 gb by the way. The program basically only runs tail recursion and a quite low amount of processes.
If you get this error while running you application that means that one of your functions are calling recursively and trying to allocate that much memory where OS would not provide to VM and hence VM crashes with that memory allocation error.
Previously when I was running into similar dumps, it was caused by a huge mailbox in a process, it had piled millions of messages.
You could check it with this snippet of code:
top() ->
Procs = lists:foldl(fun(Pid, Acc) ->
case erlang:process_info(Pid, message_queue_len) of
{_K, V} -> [{Pid, V} | Acc];
_ -> Acc
end
end, [], erlang:processes()),
lists:keysort(2, Procs).
When reading rabbitmq's rabbit.erl，it contain hipe compilation related code.
hipe_compile() ->
Count = length(?HIPE_WORTHY),
io:format("HiPE compiling: |~s|~n |",
[string:copies("-", Count)]),
T1 = erlang:now(),
PidMRefs = [spawn_monitor(fun () -> [begin
{ok, M} = hipe:c(M, [o3]),
io:format("#")
end || M <- Ms]
end) ||
Ms <- split(?HIPE_WORTHY, ?HIPE_PROCESSES)],
[receive
{'DOWN', MRef, process, _, normal} -> ok;
{'DOWN', MRef, process, _, Reason} -> exit(Reason)
end || {_Pid, MRef} <- PidMRefs],
T2 = erlang:now(),
io:format("|~n~nCompiled ~B modules in ~Bs~n",
[Count, timer:now_diff(T2, T1) div 1000000]).
But there is no explanation about hipe in the erlang's reference doc. What's the meaning of 'o3'?
(emacs#chen-yumatoMacBook-Pro.local)51> hipe:c(xx_reader,[o3]).
{ok,xx_reader}
After I use hipe:c as above, No new compiled file can be found the in the pwd() directory?
Where it is?
o3 indicates the optimization level used by the compiler. There're also levels o0, o1, o2. Details of the levels are as follows:
o1 = [inline_fp,pmatch,peephole],
o2 = [icode_range,icode_ssa_const_prop,icode_ssa_copy_prop,icode_type,
icode_inline_bifs,rtl_lcm,rtl_ssa,rtl_ssa_const_prop,spillmin_color,
use_indexing,remove_comments,concurrent_comp,binary_opt] ++ o1,
o3 = [{regalloc,coalescing},icode_range] ++ o2.
You can use hipe:help_option(Option) to further investigate the meanings of different options. For example,
3> hipe:help_option(regalloc).
regalloc - Select register allocation algorithm. Used as {regalloc, METHOD}.
Currently available methods:
naive - spills everything (for debugging and testing)
linear_scan - fast; not so good if few registers available
graph_color - slow, but gives OK performance
coalescing - slower, tries hard to use registers
optimistic - another variant of a coalescing allocator
ok
4> hipe:help_option(icode_range).
icode_range - Performs integer range analysis on the Icode level
ok
I think HiPE is JIT compilation, just as the one used in Java. The native parts are available only in runtime, so there should be no explicit representation in your file system.
Also, hipe:c do require a .beam file is present. For example, if you create a test.erl with some stuff, and without compiling it to a .beam file, call hipe:c directly will lead to an error:
1> hipe:c(test, [o3]).
<HiPE (v 3.9.3)> EXITED with reason {cant_find_beam_file,test} #hipe:419
=ERROR REPORT==== 29-Nov-2012::17:03:02 ===
<HiPE (v 3.9.3)> Error: [hipe:418]: Cannot find test.beam file.** exception error: {hipe,419,{cant_find_beam_file,test}}
in function hipe:beam_file/1 (hipe.erl, line 419)
in call from hipe:c/2 (hipe.erl, line 313)
2> c(test).
{ok,test}
3> hipe:c(test, [o3]).
{ok,test}
There is some in erlang's doc. See here. But the doc is not much indeed. The index page of HiPE only updated recently.
Also, you can check some help in erlang shell.
> hipe:help().
> hipe:help_options().
> hipe:help_option(Option).