Why binary memory usage grows up on rabbitmq - memory

Question

How to find why binary memory usage grows up (depend on connections count)
i used rabbitmqctl eval '[garbage_collect(P) || P <- processes()].'
but on stats
root#Broker1:~# /usr/sbin/rabbitmqctl status | grep binary
{binary,6709401976},
root#Broker1:~# free -m
total used free shared buffers cached
Mem: 16084 4147 11937 22 161 1550
-/+ buffers/cache: 2435 13649
Swap: 4183 0 4183
root#Broker2:~# /usr/sbin/rabbitmqctl status
Status of node rabbit#Broker2 ...
[{pid,515},
{running_applications,
[{rabbitmq_web_mqtt,"RabbitMQ MQTT-over-WebSockets adapter","3.6.9"},
{rabbitmq_mqtt,"RabbitMQ MQTT Adapter","3.6.9"},
{rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter",
"3.6.9"},
{rabbitmq_stomp,"RabbitMQ STOMP plugin","3.6.9"},
{rabbitmq_management,"RabbitMQ Management Console","3.6.9"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.9"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.9"},
{rabbit,"RabbitMQ","3.6.9"},
{os_mon,"CPO CXC 138 46","2.3"},
{cowboy,"Small, fast, modular HTTP server.","1.0.4"},
{ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},
{ssl,"Erlang/OTP SSL application","5.3.6"},
{public_key,"Public key infrastructure","0.22.1"},
{cowlib,"Support library for manipulating Web protocols.","1.0.2"},
{crypto,"CRYPTO","3.4.1"},
{amqp_client,"RabbitMQ AMQP Client","3.6.9"},
{rabbit_common,
"Modules shared by rabbitmq-server and rabbitmq-erlang-client",
"3.6.9"},
{inets,"INETS CXC 138 49","5.10.3"},
{mnesia,"MNESIA CXC 138 12","4.12.3"},
{compiler,"ERTS CXC 138 10","5.0.2"},
{sockjs,"SockJS","0.3.4"},
{xmerl,"XML parser","1.3.7"},
{syntax_tools,"Syntax tools","1.6.16"},
{asn1,"The Erlang ASN1 compiler version 3.0.2","3.0.2"},
{sasl,"SASL CXC 138 11","2.4.1"},
{stdlib,"ERTS CXC 138 10","2.2"},
{kernel,"ERTS CXC 138 10","3.0.3"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:10:10] [async-threads:160] [kernel-poll:true]\n"},
{memory,
[{total,3874499056},
{connection_readers,598416},
{connection_writers,139368},
{connection_channels,774752},
{connection_other,176069624},
{queue_procs,36524016},
{queue_slave_procs,0},
{plugins,80184008},
{other_proc,0},
{mnesia,48042392},
{metrics,7245632},
{mgmt_db,74517952},
{msg_index,1751384},
{other_ets,14801208},
{binary,3379246384},
{code,28040419},
{atom,1000601},
{other_system,33438764}]},
{alarms,[]},
{listeners,
[{clustering,2000,"::"},
{amqp,5672,"::"},
{http,15672,"::"},
{stomp,61613,"::"},
{'http/web-stomp',15674,"::"},
{mqtt,1883,"::"},
{'http/web-mqtt',15675,"::"}]},
{vm_memory_high_watermark,{absolute,"12288MiB"}},
{vm_memory_limit,12884901888},
{disk_free_limit,50000000},
{disk_free,91027542016},
{file_descriptors,
[{total_limit,65435},
{total_used,9214},
{sockets_limit,58889},
{sockets_used,9002}]},
{processes,[{limit,1048576},{used,42726}]},
{run_queue,0},
{uptime,82689},
{kernel,{net_ticktime,60}}
]
total used free shared buffers cached
Mem: 16084 4517 11567 27 160 2487
-/+ buffers/cache: 1869 14215
Swap: 4189 0 4189
my memory is free!!! but after memory grows up connection to server very slow!!
Cluster: rabbit#Broker1 (change)
RabbitMQ 3.6.9, Erlang 17.3
rabbitmq.config
[
{log_levels, [{connection, none},{channel,none}]},
{file_descriptors, [{total_limit, 65000}]},
{kernel, [
{log_levels, [{connection, none},{channel,none}]},
{inet_dist_listen_min, 2000},
{inet_dist_listen_max, 65000}
]},
{rabbit, [
{loopback_users, []},
{vm_memory_high_watermark, {absolute, "12288MiB"}},
{vm_memory_high_watermark_paging_ratio, 0.60},
{tcp_listeners, [5672]},
{log_levels, [{connection, none},{channel,none}]},
{heartbeat, 10000}
]},
{rabbitmq_mqtt, [
{log_levels, [{connection, none},{channel,none}]},
{default_user, <<"guest">>},
{default_pass, <<"guest">>},
{allow_anonymous, false},
{vhost, <<"test">>},
{exchange, <<"amq.topic">>},
{subscription_ttl, 180000},
{prefetch, 10},
{tcp_listeners, [1883]},
{tcp_listen_options, [
binary,
{packet, raw},
{reuseaddr, false},
{backlog, 8192},
{nodelay, true}
]}
]},
{rabbitmq_stomp, [
{default_vhost, <<"test">>},
{exchange, <<"amq.topic">>},
{tcp_listen_options, [
{backlog, 128},
{nodelay, true}
]}
]}
].

Answer 1

I believe your implementation is incorrect. I am using RabbitMQ for live notifications on a live site and it works without any problems. I had a concurrent clients of upto 3500, but a 8GB RAM instance worked well enough.
So what you must exactly do?
You have not specified your use case. But as per rabbitmq documentations, the memory grow only when you have lot of undelivered messages. Lot of undelivered messages can cause the disk space/ram space depending on how your clients connect to you.
If you have one broker per user, that is very costly. Instead, have one exchange per user, if your user is going to receive same message on multiple ends, eg, phone and a web at the same time. In that case, use a fanout exchange and make the user subscribe to it. Always enable auto delete flag when messages in exchanges reaches zero. Use non-persistant exchanges; but keep your brokers less.
If you are sure that one user is going to connect on only one device and not more than that, you don't need a fanout exchange, instead, build one queue per user. Still, enable auto delete flags.
Keeping less number of brokers and handling requests with exchanges or queues are efficient.
One more option to use if it suits your use case is message expiry. Let's say you are using RabbitMQ to send notifications to your users. You will however write them to a database and to make a notification, you may send it via RMQ. If the user is not online, may be for about 10 min or more, the expiry is a nice option. Because it is however live messaging, not persistent messaging. He is going to read it from database
NOTE: I can help you in detail if you explain your usecase.
Read this to understand more: https://www.rabbitmq.com/blog/2014/10/30/understanding-memory-use-with-rabbitmq-3-4/
As said here,
All binary use is due to messages in queues.
Best way is to use Non Persistent queues and auto-expiry messages!
Note that "In memory" and "Persistent" messages are not antonyms here:
a non-persistent message can be paged out under memory pressure and a
persistent message can be in memory too.
That explains your issue!