How many times per minute does the MQTT client poll the server? Is it a big data traffic or not? I know the size of the packet can be small, but how many times the client ping the broker to make itself "online" in the broker.
If I was not clear please comment this question and I'll try explain better my doubt.
My broker is Mosquitto and the clients are small device (sensors and etc.)
Assuming no data flow (which is of course application dependent), the client will periodically send a PINGREQ message to the broker. This is a 2 byte message and the broker replies with a PINGRESP, also 2 bytes.
The rate at which PINGREQ is sent depends on the keepalive parameter set when you connect. This tells the broker the interval at which it should expect at least one message from the client. In the absence of any other message, the client sends a PINGREQ.
60 seconds is often used as a default value (whether or not this is appropriate for you depends on how quickly you want the client/broker to respond to a hung connection). In the absence of any other messages flowing, maintaining the keepalive guarantee would mean 4 bytes total transferred every minute. This is only the application level data of course, the length of the data on the wire will be bigger.
Related
According to the MQTT specification, a QoS 2 message sent by a MQTT client must follow this workflow:
During the various phases Mosquitto stores the message in its memory. This is also confirmed by looking at the mosquitto.db persistent storage using the db_dump tool described here.
The question is: if a malicious client PUBLISH tons of messages with QoS 2 but never sends the PUBREL message as a response to PUBREC what happens ? Mosquitto keep the messages undefinitively ? I expected some kind of configuration parameter able to get rid of such unacknowledged messages after some time but I can't find any.
I'm not 100% sure, but I think the max_inflight_messages setting should kick in here and not allow the client to send the second QOS 2 message until the first has been completed.
This would limit each client to one malicious message at time.
There are a number of libraries that will allow you low level control over when packets are sent so building a PoC shouldn't be that hard. And if it is possible to trigger a DoS style attack, I'm sure eclipse/mosquitto would look kindly at a Pull Request with a fix.
I wrote an MQTT client program, which runs at a computer (computer 1). The MQTT client program connects to an MQTT Broker with QoS=1 and publishes data to Broker periodically. I subscribe the Broker (Qos=1) from another computer (computer 2), using Mosquito utility. I found the data published to Broker will be delay delivered to publisher about 3 seconds. The delayed time is too long. I checked the codes and found the 3-second-delay time is from read_packet() which is to read back acknowledge from Broker. Why is there such long delay time? How can I figure it out? The Broker (MQTT server) is managed by my coworker. If the Broker is the cause, I can request them to help. But I need to know what could be the trouble source, so that I can check with them.
I can confirm the delay occurring at the time of reading back acknowledge from Broker by watching the debugging message from MQTT client program at computer 1. For Qos = 1, the client must read back acknowledge after sending (publishing) packets. I found 3-second delay time between sending packet and reading back acknowledge. Surely, I also found the delay at display of Mosquito_sub utility.
Assuming near instant network comms and nothing else strange going on the fact that you have recreated the problem with mosquitto_sub then this points to the MQTT broker being the source of the problem.
Without knowing what broker you are using and how heavily it is loaded it's hard to say more but you should look at the broker logs.
I want to find the queuing time in MQTT queues such as arrival time as the event enters the queue and the ingestion time when the event is taken from the queue. Subtracting both these times can give me queueing delay. How do I find it?
There is no queuing in MQTT (apart for offline clients with high QOS subscriptions), messages are delivered to all clients subscribed to a topic as soon as it is received by the broker.
If you want to know how long it takes a broker to process a new message then it will depend on the broker, the machine it's running on, the number of clients subscribed to the topic (and at what QOS) and what the load level is. You may be able to calculate this by increasing the logging on your given broker, but this will be broker specific and any increase in logging level is likely to increase the latency as well. Your best bet would be to look at the network traffic to track inbound and outbound messages, something like wireshark would probably be best.
I wonder how much data will be consumed (approximately) over say one month if I just connected to an MQTT server.(without sending or receiving any messages).
I need to calculate it to measure what data plan should I recharge for my sim card used in IoT application.
Thanks
Edit: fix broken link as suggested by #Jacob
I hope that you can find your answers here:
MQTT data usage
By the way... The client sets the keep alive value when it sends a CONNECT request to the server (aka the broker).
Potentially, if the client choose 0 as keep alive value, no data is consumed except for setup connection.
Last time I had to do something similar I simulated the traffic of a month.
Because computers are much faster than IoT devices, just use any kind of high-level library to send the traffic the IoT devices would be sent in a month and then measure the TCP traffic.
The packet identifier is required for certain MQTT control packets (http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/csprd02/mqtt-v3.1.1-csprd02.html#_Toc385349761). It's defined by the standard as a 16bit integer, and is generated by each client. The identifiers are reusable by the client after the acknowledgement packet is received. So the standard allows up to 64k in-flight messages. In practice, the clients I've looked at seem to just increment a counter, and so allow a total of 64k messages to be sent by a client. Both of rust MQTT client libraries panic when that counter overflows. (UPDATED 2016-09-07: if the rust clients are compiled in release mode then they don't panic, the value of the Packet Identifier becomes 0 -- in normal circumstances this will work, but...)
Does anyone know of an MQTT client that allows more than 64k messages/client (i.e. re-uses packet identifiers)? I'm wondering if this is a limitation that I need to be aware of in general, or if it's just a few clients. I've taken a quick look at compliance tests and haven't yet seen much to indicate that this is checked -- I'll keep looking.
Edit: It could be that some clients achieve this as a side-effect of limiting the number of in-flight messages. UPDATE 2016-09-07 the rust clients do it by assuming they can wrap on overflow and never catch up to lagging messages (maybe a good bet, but not assured, and with an ugly outcome if it happens)
As you have pointed out, the packet identifier are intended as temporary value that must persist until the published packet is received and acknowledged.
Once acknowledged, you can reuse the identifier (or not).
Most client runs on embedded system and they don't track more than a single packet (so only a single identifier is being handled) since they wait for ACK or REC/COMP before making any other publishing.
So for these clients, even a single identifier would be enough.
Please notice that for QoS 1, remembering the identifier is futile since it's valid to resend the packet if the next packet is not an ACK (so you have the identifier to reply with in the packet you are receiving).
For the rare clients that do support interleaved publish packets, they only need to support 2 valid identifiers at any time (that is, in the case they have received a QoS 2 packet, answered with PUBREC and then receive another QoS 1 or 2 packet).
Once they receive a PUBREL packet they can reply with a PUBCOMP without needing to remember the identifier (it's in the PUBREL header), so the only time they do need to remember identifier is between the PUBLISH and the PUBREC packet. Provided they allow interleaved publish packets, the only case where a second identifier is required is when they are publishing while receiving a published packet at the same time.
Now, from the point of view of the broker, most implementation use a 16-bit counter per client so they could support, in theory, up to 65535 in-transit packets.
In reality, since the minimum size of a publish packet is 8 bytes (usually more), that means having to store at least 9 bytes per potential packet (the additional byte is for remembering the current state in case of QoS 2) so that's half a MB of memory in the minimal case, but likely much more in real life, since you never have an empty publish payload and topic name.
As you see, it's almost impossible for an embedded system to implement with such storage requirement so shortcut are taken.
In most scenario, either the server does not allow so many un-acknowledged packet (by simply replying to the client in order to release the identifier) or use the identifiers pool between different clients.
So typically, again, the worst case for the broker can only happen if the client does not acknowledge the published packets. If the broker does not get any answer from the client it can:
close the connection
refuse to send new published information or
ignore the answer and republish
All of these strategies needs to be implemented anyway since you could have the same issue with a slow client and a fast publisher and your 65535 identifiers.
Since you have these strategies, there is no need to waste a MB of memory per client and instead cut your leg much earlier (while keeping a reasonable working condition).
In the end, the packet identifiers are a tool to deal with identification of recent packets, not a tool to index all packet received. A counter is good enough for this case and a wrapping around should not pose any issue when you account for the memory and bandwidth requirement.