My server keeps getting filled up with ejabberd crash logs every few hours - it seems that the ejabberd server keep crashing and the crash log will fill the server's free space until the server has no more space left (GBs of crash logs). The crash logs start with something like this:
=erl_crash_dump:0.1
Tue Feb 4 23:44:02 2014
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,kernel, {shutdown,{kernel,start,[normal,[]]}}})
System version: Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]
Compiled: Fri Dec 16 03:22:15 2011
Taints:
Atoms: 4574
Can anyone see something from the crash log and let me know what's happening?
In this case, the crash dump is unlikely to tell you very much - it tells you that the kernel application shut down ({shutdown,{kernel,start,[normal,[]]}}), but it doesn't say why. In the error log you should find a number of crash reports and error messages that led to the node shutting down.
Crash dumps are much more useful if the node crashes because it runs out of memory. In that case, you can usually see which process behaved badly.
Related
We have an erlang application that crashed due to memory issue:
Thu Jun 9 13:22:19 202
Slogan: eheap_alloc: Cannot allocate 3936326656 bytes of memory (of type "heap").
System version: Erlang/OTP 23 [erts-11.1.1] [source] [64-bit] [smp:48:48] [ds:48:48:10] [async-threads:1] [hipe]
Compiled: Mon Oct 12 07:40:16 2020
The total memory in =memory section of the crash dump says:
=memory
total: 126190588240
processes: 99882909648
processes_used: 99879758664
system: 26307678592
atom: 6308969
atom_used: 6281592
binary: 221919896
but when I total the processes' memory in the =proc section, it turns out to be 24660309644 bytes which is way less than the above mentioned for processes_used i.e 99879758664 bytes.
Can anyone help me understand why and where could this difference be?
I followed the tutorial many times and, created a small app using rebar and release it successfully. But when I come to the main application (calendarApp), crash dump was returned. I made some changes but the result is the same. I would like to know what I am doing wrong, and how I can fix this.
At the end, when I do start and attach, it returns "Node is not running!".
I will show you:
sonu#sonu-Ideapad-Z570:~/calendarApp/rel$ ./calendarApp/bin/calendarApp console
Exec: /home/sonu/release/calendarApp/rel/calendarApp/erts-5.10.4/bin/erlexec -boot /home/sonu/release/calendarApp/rel/calendarApp/releases/1/calendarApp -mode embedded -config /home/sonu/release/calendarApp/rel/calendarApp/releases/1/sys.config -args_file /home/sonu/release/calendarApp/rel/calendarApp/releases/1/vm.args -- console
Root: /home/sonu/release/calendarApp/rel/calendarApp
Erlang R16B03 (erts-5.10.4) [source] [smp:4:4] [async-threads:10] [kernel-poll:false]
{global,calendarApp_sup} (<0.216.0>) starting....
{global,calendarApp_server} (<0.217.0>) starting ......
Eshell V5.10.4 (abort with ^G)
(calendarApp#127.0.0.1)1>
=INFO REPORT==== 19-Jan-2016::09:50:45 ===
application: calendarApp
exited: {{shutdown,
{failed_to_start_child,calendarApp_server,
{undef,
[{mnesia,create_schema,
[['calendarApp#127.0.0.1']],
[]},
{calendarApp_db,init,0,
[{file,"src/calendarApp_db.erl"},{line,7}]},
{calendarApp_server,init,1,
[{file,"src/calendarApp_server.erl"},{line,49}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}}},
{calendarApp_app,start,[normal,[]]}}
type: permanent
{"Kernel pid terminated",application_controller," {application_start_failure,calendarApp,{{shutdown,{failed_to_start_child,calendarApp_server,{undef,[{mnesia,create_schema,[['calendarApp#127.0.0.1']],[]},{calendarApp_db,init,0,[{file,\"src/calendarApp_db.erl\"},{line,7}]},{calendarApp_server,init,1,[{file,\"src/calendarApp_server.erl\"},{line,49}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,239}]}]}}},{calendarApp_app,start,[normal,[]]}}}"}
Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,calendarApp,{{shutdown,{failed_to_start_child,calendarApp_server,{undef,[{mnesia,create_schema, [['calendarApp#127.0.0.1']],
sonu#sonu-Ideapad-Z570:~/calendarApp/rel$ ./calendarApp/bin/calendarApp console
If anyone knows the problem, please help me to fix this.
When I am trying to install yaws 1.89, the error bellow is raised. Please help me to overcome that error.
~/yaws$ sudo yaws
Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:true]
Eshell V5.10.4 (abort with ^G)
1> =ERROR REPORT==== 14-Mar-2014::15:05:09 ===
Failed to load setuid_drv (from "/usr/local/lib/yaws/priv/lib") : "Driver compiled with incorrect version of erl_driver.h"
=ERROR REPORT==== 14-Mar-2014::15:05:09 ===
FATAL {'EXIT',normal}
=INFO REPORT==== 14-Mar-2014::15:05:09 ===
application: yaws
exited: {{shutdown,
{failed_to_start_child,yaws_server,
{badconf,
[{yaws_server,init,1,
[{file,"yaws_server.erl"},{line,159}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}}},
{yaws_app,start,[normal,[]]}}
type: permanent
{"Kernel pid terminated",application_controller,"{application_start_failure,yaws,{{shutdown,{failed_to_start_child,yaws_server,{badconf,[{yaws_server,init,1,[{file,\"yaws_server.erl\"},{line,159}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,239}]}]}}},{yaws_app,start,[normal,[]]}}}"}
Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,yaws,{{shutdown,{failed_to_start_child,yaws_server,{badconf,[{yaws_server,init,1,[{file,"yaws_server.erl"},{line,159}]},{ge
Yaws 1.89 was released in September 2010, but you're trying to run it on a newer version of Erlang, R16B03, which was released in December 2013. The drivers Yaws uses, which are compiled native code, were compiled using a version of the Erlang driver interface that doesn't match that of the Erlang version you're running, which is what causes the error messages you're seeing.
Your comment above hints that you were able to get it working by downloading yaws-1.89.tar.gz and building it yourself; if so, then yes, that's a good way to get it working with your current version of Erlang. But a better way would be to use a newer version of Yaws — the latest version at this time of writing is Yaws 1.98, released in November 2013.
I get this message when launching RabbitMQ:
=WARNING REPORT==== 8-Feb-2014::10:43:42 ===
Only 2048MB of 23482MB memory usable due to limited address space.
Crashes due to memory exhaustion are possible - see
http://www.rabbitmq.com/memory.html#address-space
When I follow that link, I read about how I should be using a 64-bit Erlang VM. But:
ajax:~ maxvitek$ erl
Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Eshell V5.10.4 (abort with ^G)
1>
...which certainly appears to be a 64-bit build. This is with the vm_memory_high_watermark set to 1. If I can get rid of the memory address problem so that RabbitMQ could use more of the system's memory, I will set that back to 0.4. Any idea where to look to fix this?
Both Erlang and RabbitMQ are installed via Homebrew, running on Mavericks.
I am not able to understand the reason why Erlang crashes and restarts. I am running Ejabberd server and its log folder is always full of erl_crash_xxxx.dump files. How can I debug this problem.
Here is a small part of erlang.log file:
=CRASH REPORT==== 4-Sep-2013::19:44:51 ===
crasher:
initial call: ejabberd_http:init/2
pid: <0.15614.15>
registered_name: []
exception exit: {normal,
{gen_fsm,sync_send_all_state_event,
[<0.15454.15>,
{http_put,2020093061,
[{"xmlns",
"http://jabber.org/protocol/httpbind"},
{"rid","2020093061"},
{"sid",
"26820e4cd7d331de864b857d1ef3351caf7dbac5"}],
[],115,1,[],
{{49,205,148,16},56132}},
30000]}}
in function gen_fsm:sync_send_all_state_event/3
in call from ejabberd_http_bind:http_put/7
in call from ejabberd_http_bind:handle_http_put/7
in call from ejabberd_http:process/2
in call from ejabberd_http:process_request/1
in call from ejabberd_http:process_header/2
in call from ejabberd_http:receive_headers/1
ancestors: [ejabberd_http_sup,ejabberd_sup,<0.37.0>]
messages: []
links: [<0.274.0>,#Port<0.1519795>]
dictionary: []
trap_exit: false
status: running
heap_size: 2584
stack_size: 24
reductions: 1082
neighbours:
These are the top few lines of a typical crash dump file:
=erl_crash_dump:0.1
Tue Sep 3 16:31:47 2013
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}})
System version: Erlang R14B04 (erts-5.8.5) [source] [64-bit] [rq:1] [async-threads:0] [hipe] [kernel-poll:false]
Compiled: Wed Oct 5 17:25:18 2011
Taints:
Atoms: 4699
=memory
total: 21498768
processes: 556368
processes_used: 541208
system: 20942400
atom: 322177
atom_used: 302233
binary: 18216
code: 2165726
ets: 53736
=hash_table:atom_tab
size: 3203
used: 2471
Those 'crashes' are probably totally unrelated. What you see as '=CRASH REPORT=' in your log files are sort of 'normal' or expected crashes and as such handled by a supervisor. The one you posted is a crash inside a handler for a HTTP call while creating or sending the a response that might ended up in some unforseen condition (like when the sending process does not exist anymore - not sure about that though). Erlang is built and designed to handle such crashes gracefully. In other words, your ejabberd is still running and serving requests happily.
The other one is a crash dump which results from a real crash of the Erlang runtime. This might happen if for example the machine hosting the runtime runs out of memory. In your case it seems like a configuration error, the node could not boot correctly at all. See http://www.ejabberd.im/node/872.