I am doing a request/response flow using a MQTT broker and I wondered if brokers like VerneMQ or Mosquitto deal well with huge amount of topics. Basically every time I want to do a request/response, I publish to a topic that looks like rpc/{UUID} meaning every request creates a new topic and then unsubscribe from it when the response is received. Will this come and bite me later ?
Topics are effectively ephemeral already.
Usually the only overhead to a topic is in the list of subscribed topic patterns (because they can be wildcards) held for each client. The topic is read from an incoming messages and checked against this list.
Using UUIDs in topics should not cause any problems.
Related
So I am using the Mosquito MQTT broker for a project and I'm subscribing to a topic that has a variable number of subtopics. Is there any way of knowing how many subtopics a topic has without just getting the update of all the subtopics first?
No
Topics are ephemeral, they don't really exist except at the instant a message is published to one.
Subscribing clients supply a pattern (as it can include wildcards that match 1 or many levels of a topic + or #) and the broker matches that against the topic in the published message to decide if it should forward it to the subscriber.
The only time the broker keeps track of a topic is if it is storing a message with the retained flag set, or queuing a message for an offline client with a persistent subscription.
I would like read messages from a MQTT broker ordered (chronologically as written) from more than one topic. For example I have topics (which are published to independently by clients at different rates all with QoS 2):
/foo/a
/foo/b
/foo/c
The messages are in a Persistent Session for a long period using Message Expiry Interval and the subscriber could come and go, on and offline, with any number of messages on each topic not yet read.
When I subscribe to: /foo/#, will I receive messages from topics /foo/a interleaved with messages from /foo/b and /foo/c in the order they were received by the broker ?
The specification on Message Ordering says:
... When a Server processes a message that has been published to an Ordered Topic, it MUST send PUBLISH packets to consumers (for the same Topic and QoS) in the order that they were received from any given Client [MQTT-4.6.0-5 ...
"(for the same Topic and QoS)" suggests ordering can only be guaranteed for the same Topic and QoS. So the answer my question of ordering across topics seems to be undefined.. ?
From the Mosquitto broker point of view, if a client has disconnected but has a long session expiry time and has active subscriptions with QoS>0, that is no different to a client that is connected - the session remains open. That means the messages will be delivered according to the ordering requirements in the spec.
This part of the answer covers retained messages only:
My understanding is that message ordering rules only apply for active sessions. That is to say, a client publishes messages and they must be delivered to current consumers only in the same order they were received.
It does not, however, apply to the situation when a client subscribes to a topic filter and receives retained messages. You can get a clue to the intent of the spec there, because the concept of messages being out of order for the same topic and QoS is nonsensical when there is only a single retained message per topic.
Ordering of delivery of retained messages that match a wildcard subscription is undefined. In Mosquitto is it roughly in order of delivery, breadth then depth. This is likely to change in the future to being sorted though.
Would the mqtt community consider placing message information in the topic name an anti-pattern?
I have a client that has a vast library written around rabbitmq, and I'm trying to tweak their client and server code to allow them to configure their services for mosquitto instead. One central requirement for them is TTL, the clients can sometimes sit for hours publishing data before the server comes back online and they do not want messages to show up that are beyond their TTL.
Their message envelope system is an elaborate json and 1) it would be painful to wrap or alter this json 2) I do not want to incur the expense of unmarshalling json to retrieve a timestamp.
The easiest thing to do is place the timestamp at the end of the topic and consume with wildcards: mytopic/mysubtopic/{timestamp} consumed by mytopic/mysubtopic/#
Are there any unintended consequences for this, and would this be considered an anti-pattern?
Whether this is an anti-pattern is a matter of opinion; the spec defines the topic as "The label attached to an Application Message..." so does not preclude your usage. I can think of a few potential "unintended consequences" to your approach (which may, or may not, apply to your specific situation):
Retain flag: As per your comment you will not be able to set the Retain flag to 1 (because all messages would be retained).
Latest Message only when comms re-established: A subscriber may only want the latest message when communications are re-established. This can be achieved by publishing messages with the retain flag set to 1 which results in your subscriber receiving the latest message (and only the latest message; subject to QOS/CleanSession) on each topic it subscribes to (docs). As per the above this will not work with your topic structure.
Order of delivery: the spec requires that "A Server MUST by default treat each Topic as an 'Ordered Topic'" but there is no such guarantee across topics. Note that ordered delivery is dependent upon settings (see the "Non normative comment" in the spec) so this may not be an issue.
Topic Alias: MQTT V5 introduces Topic Alias which can be used to reduce the amount of data transmitted. This will not provide a benefit with your structure.
I implemented a MQTT message broker using mosquitto on my network. I have one web app publishing things to the broker and several servers that subscribed the same topic. So i have a redundancy scenario.
My question is, using mosquitto alone, is there any way to configure it to publish data only on the first subscriber? Otherwise, all of them will do the same thing.
I don't think that is possible.
But you can do this.
Have the first subscriber program respond with an ack on the channel as soon as it gets the message, and have the redundancy program look for the ack for a small time after the initial message.
IF the ack is received the redundancy should not do anything.
So if the first subscriber gets and uses the message, the others wont do anything even if they get the message.
No this is not possible with mosquitto at the moment (without communication between the 2 subscribers as described in the other answer).
For the new release of the MQTT spec (v5)* there is a new mode called "Shared Subscriptions". This allow s multiple clients to subscribe to a single topic and messages will be delivered by round robin to each client. This is more for load balancing rather than master/slave fail over.
*There are some brokers (HiveMQ, IBM MessageSight) that already support some version of Shared Subscriptions at MQTT v3.1.1, but they implement it in slightly different ways (different topic prefixes) so they are not cross compatible.
I want to display delivered and read receipts to users in my messaging platform. I am using Eclipse's Paho library with Mosquitto as the broker. Since Mosquitto does not store messages, which is the best way/plugin to
Display delivered receipts - how to use QoS2 acknowledgement receipts to do this?
Display read receipts - suggest me way to do this
How to store messages so that users can view their chat history? Any architectural insights in mysql will be very helpful.
The quick answers to your questions:
High QOS (1/2) is not end to end delivery confirmation, it is only confirmation between the broker and a client. e.g. a publisher publishing at QOS 2 the confirmation is only between the publisher and the broker, not then onward to the subscriber (who may be subscribed at a different QOS anyway). The only way to do this is to send a separate message from the receiving end back to the sender. Also there may be more than one subscriber to any given topic, so you have to think how this would work.
Again, the only way to do this is with a separate message sent when the message is read
You will have to implement this yourself. The only thing that may help is something like the built in support for storing messages in a database present in some brokers (this is not part of the spec, so totally propitiatory to the implementation) e.g. hivemq
Hardlib's answers are 100% on target, but I'll add some thoughts on implementation.
I think a common misunderstanding with MQTT is that it is really a M2M (machine-to-machine) protocol, not a system for exchanging messages between users. That isn't to say you can't use it for messaging (facebook did) but that exists in a layer on top of MQTT. Put another way, MQTT is designed to route messages between machines with little care about the content of those messages. What this means is that user-level niceties (delivery confirmations etc.) aren't really part of it but instead something that you implement on top of MQTT.
So here are some thoughts about how to implement what you propose on top of MQTT:
Consider a situation in which you have two clients (X & Z) which both have access to the same broker (Y). To have client X confirm it has received a message from client Z, simply have client X send a message to a topic (lets say confirmations/z) that client Z is subscribed to. This is trivial to implement in Python or whatever you are writing your application in. (For example, I use that basic procedure to measure round-trip time on my broker.)
However, given that QoS can guarantee that a broker has received the message (and it could be retained or otherwise held for other clients), I would question if this is really necessary unless it is critical that client Z know exactly when client X receives the message.
Depending on your needs there are any number of ways of providing a history for a topic. See the answers here and here for details on MySQL. But if all you need is a local history of a chat or a record of the activity on a few topics, consider simply outputting payloads with timestamps to a text file or JSON. MySQL feels like overkill unless you are dealing with a high volume of messages or need to compose complex queries.