Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
With a budget of perhaps a few million to setup a MQTT server farm how would you do so?
It must have the following properties:
Support for 4-5M connections across all data centers.
300k msg/s of around 1kb each
Geographic redundancy
Can't lose messages QOS1
Each client will publish to a single topic but subscribe to their own unique topic. This implies 4-5 million topics.
MQTT Server brokers can be found here:
https://github.com/mqtt/mqtt.github.io/wiki/server-support#capabilities
However capabilities are usually not published.
Although the Erlang powered VerneMQ MQTT broker is still quite new, there is nothing (besides RAM/CPU/IPs/Bandwidth) that should prevent you from opening that many connections.
http://verne.mq
make sure to set something similar to:
listener.max_connections = infinity
listener.nr_of_acceptors = 1000
erlang.max_ports = 10000000
erlang.process_limit = 10000000
in your vernemq.conf
disclaimer: I'am one of the devs of VerneMQ and happy to help you reach your 1M connections/server.
HiveMQ is a self hosted, java based Enterprise MQTT broker, specifically designed to support millions of concurrent connections.
The HiveMQ team has done a benchmark, connecting more than 10.000.000 concurrent MQTT clients to a HiveMQ broker cluster. To reach this number, accompanied with a decent performance, some configuration needed to be done on the operating system of the used machines.
Open files in /etc/security/limits.conf:
hivemq hard nofile 1000000
hivemq soft nofile 1000000
root hard nofile 1000000
root soft nofile 1000000
TCP tweaking in /etc/sysctl.conf
# This causes the kernel to actively send RST packets when a service is overloaded.
net.ipv4.tcp_fin_timeout = 30
# The maximum file handles that can be allocated.
fs.file-max = 5097152
# Enable fast recycling of waiting sockets.
net.ipv4.tcp_tw_recycle = 1
# Allow to reuse waiting sockets for new connections when it is safe from protocol viewpoint.
net.ipv4.tcp_tw_reuse = 1
# The default size of receive buffers used by sockets.
net.core.rmem_default = 524288
# The default size of send buffers used by sockets.
net.core.wmem_default = 524288
# The maximum size of received buffers used by sockets.
net.core.rmem_max = 67108864
# The maximum size of sent buffers used by sockets.
net.core.wmem_max = 67108864
# The size of the receive buffer for each TCP connection. (min, default, max)
net.ipv4.tcp_rmem = 4096 87380 16777216
# The size of the sent buffer for each TCP connection. (min, default, max)
net.ipv4.tcp_wmem = 4096 65536 16777216
Details on the VMs used, the specific configurations needed on the OS side and detailed performance results can all be found in the 10 Million Benchmark Paper
Disclaimer: I am part of the HiveMQ Team.
IBM Messagesight appliance. Specifically designed for large scale IOT deployments such as connected cars:
http://www-03.ibm.com/software/products/en/messagesight
Clustering IBM IoT MessageSight servers is possible with v2.0 which allows you to scale a single MessageHub to multiple servers, thus enabling >1M connections even.
Akiro MQTT Broker deals with this scale and is a very reliable and low latency broker which is available which is powered by Async IO.
Akiro can handle 10 Million connections with 12 brokers with commodity hardware, which is one of the best benchmarks for a MQTT broker today. Its also being used by major telecoms. Give it a shot. Thanks
P.S I am part of the Akiro team :)
You do not need few million dollars to achieve this. Actually you do not need even tens of thousands - flespi broker in it's commercial version achieves all numbers that you (were) need, except Geographic redundancy at that moment of time. And not only achieves - it is being used with multiple similar loads each day, 24/7 with 99.98% uptime.
It is cloud based broker with private namespace, so even it's free version available for everybody capable of serving the traffic up to 200 MB/minute.
Related
My question is if I plan to use WebRTC with a p2p architecture but only using its custom data channel to send constant small text messages. ¿What is the maximum number of peer connections that a peer can support? (I know this its heavily going to depend on the device, network... of each peer, but could somebody give me a ballpark estimate).
Edit: By constant text messages i mean around 30 / sec
One of limitations might be maximum amount of available ports in the device's OS. For example, Ubuntu has about 65k available ports. So, supposing that you have enough memory, CPU and network bandwith, and 1 port for 1 data channel then you have ~65k connections.
We operate two dual-node brokers, each broker having quite different queues and workloads. Each box has 24 cores (H/T) worth of Xeon E5645 # 2.4GHz with 48GB RAM, connected by Gigabit LAN with ~150μs latency, running RHEL 5.6, RabbitMQ 3.1, Erlang R16B with HiPE off. We've tried with HiPE on but it made no noticeable performance impact, and was very crashy.
We appear to have hit a ceiling for our message rates of between 1,000/s and 1,400/s both in and out. This is broker-wide, not per-queue. Adding more consumers doesn't improve throughput overall, just gives that particular queue a bigger slice of this apparent "pool" of resource.
Every queue is mirrored across the two nodes that make up the broker. Our publishers and consumers connect equally to both nodes in a persistant way. We notice an ADSL-like asymmetry in the rates too; if we manage to publish a high rate of messages the deliver rate drops to high double digits. Testing with an un-mirrored queue has much higher throughput, as expected. Queues and Exchanges are durable, messages are not persistent.
We'd like to know what we can do to improve the situation. The CPU on the box is fine, beam takes a core and a half for 1 process, then another 80% each of two cores for another couple of processes. The rest of the box is essentially idle. We are using ~20GB of RAM in userland with system cache filling the rest. IO rates are fine. Network is fine.
Is there any Erlang/OTP tuning we can do? delegate_count is the default 16, could someone explain what this does in a bit more detail please?
This is difficult to answer without knowing more about how your producers and consumers are configured, which client library you're using and so on. As discussed on irc (http://dev.rabbitmq.com/irclog/index.php?date=2013-05-22) a minute ago, I'd suggest you attempt to reproduce the topology using the MulticastMain java load test tool that ships with the RabbitMQ java client. You can configure multiple producers/consumers, message sizes and so on. I can certainly get 5Khz out of a two-node cluster with HA on my desktop, so this may be a client (or application code) related issue.
In the WebSocket++ 0.3.x library, what determines the limit of how many WebSocket clients can have an active connection? Is it one connection per thread, or can one thread handle multiple WebSocket client connections? If it is the latter, roughly how many connections can one thread hold?
Basically, I'm looking for a ballpark number of how many client connections WebSocket++ library can handle in a application with roughly 25 threads to spare. The library homepage is:
http://www.zaphoyd.com/websocketpp
If you are using the Boost.Asio based transport policy with a recent version of Boost on a platform that supports non-blocking/asynchronous I/O (epoll on Linux, kqueue on OS X/BSD, iocp on Windows) then WebSocket++ does not introduce any significant limits on simultaneous connections.
In such situations, the limits are pretty much based on OS, hardware, and application factors. The OS will limit total file descriptors in use per process (with root access this limit can be changed). High concurrency levels will require your application to be structured appropriately to handle it (primarily using short, bounded time, non-blocking handlers). Other factors will limit you in the same way that generic servers will be limited. Gigabit Ethernet can only handle so much traffic, using TLS or compression will reduce performance, etc.
I haven't done extensive performance benchmarking with 0.3.x yet, but 0.2.x in appropriately tuned applications were able to easily service 10s of thousands of concurrent clients on an i7 core.
It is the intention of the WebSocket++ architecture to scale to arbitrary connection counts given sufficient resources. If you are working on an application that scales WebSocket++ beyond 10k connections I'd be interested in more details and in addressing any bottlenecks you discover.
I'm using CentOS (Linux) and was wondering
the maximum connection that one server can have through epoll (Edge Trigger, OneSHot) .
I've succeeded in having 100,016 connections doing ping-pongs (nonstop) atm. How many socket connections can one server handle?
I don't think it is unlimited. If anyone who've tried it. Could you please share ?
500,000 TCP connections from a single server is the gold standard these days. The record is over a million. It does require kernel tuning. See, for example, Linux Kernel Tuning for C500k.
Does the erlang TCP/IP library have some limitations? I've done some searching but can't find any definitive answers.
I have set the ERL_MAX_PORTS environment variable to 12000 and configured Yaws to use unlimited connections.
I've written a simple client application that connects to an appmod I've written for Yaws and am testing the number of simultaneous connections by launch X number of clients all at the same time.
I find that when I get to about 100 clients, the Yaws server stops accepting more TCP connections and the client errors out with
Error in process with exit value: {{badmatch,{error,socket_closed_remotely}}
I know there must be a limit to the number of open simultaneous connections, but 100 seems really low. I've looked through all the yaws documentation and have removed any limit on connections.
This is on a 2.16Ghz Intel Core 2 Duo iMac running Snow Leopard.
A quick test on a Vista Machine shows that I get the same problems at about 300 connections.
Is my test unreasonable? I.e. is it silly to open 100+ connections simultaneously to test Yaws' concurrency?
Thanks.
It seems you hit a system limitation, try to increase the max number of open files using
$ ulimit -n 500
Python on Snow Leopard, how to open >255 sockets?
Erlang itself has a limit of 1024:
From http://www.erlang.org/doc/man/erlang.html
The maximum number of ports that can be open at the same time is 1024 by default, but can be configured by the environment variable ERL_MAX_PORTS.
EDIT:
The system call listen()
has a parameter backlog which determines how many requests can be queued, please check whether a delay between requests to establish connections helps. This could be your problem.
All Erlang system limits are reported in the Erlang Efficiency Guide:
http://erlang.org/doc/efficiency_guide/advanced.html#id2265856
Reading from the open ports section:
The maximum number of simultaneously
open Erlang ports is by default 1024.
This limit can be raised up to at most
268435456 at startup (see environment
variable ERL_MAX_PORTS in erlang(3))
The maximum limit of 268435456 open
ports will at least on a 32-bit
architecture be impossible to reach
due to memory shortage.
After trying out everybody's suggestion and scouring the Erlang docs, I've come to the conclusion that my problem is with Yaws not being able to keep up with the load.
On the same machine, an Apache Http Components web server (non-blocking I/O) does not have the same problems handling connections at the same thresholds.
Thanks for all your help. I'm going to move on to other erlang based web servers, like Mochiweb.