Assuming I have topic A and topic B.
Upon starting my application, inside on_connect, I am subscribing to both topics with QOS 2. Then I start the thread with loop_forever().
Now, lets assume I have missed messages on topic B which were sent while I was gone and I have retained messages on topic A.
Upon subscribing, which messages will be processed first?
Is it defined due to the type (i.e. persistent session messages and retained messages on a topic)?
Or is the subscription order in on_connect decisive? (which I would not assume as the thread starts checking them upon loop_forever() only and the order might not be presumed)
Or is it random?
From my tests, it seems the missed messages I get due to the persistent session from topic B will be processed first.
Can I rely on that behaviour?
PS: As hardillb mentioned that behaviour might be broker specific. I am using Mosquitto.
First your model is a little wrong, the client doesn't check for anything, message delivery is entirely driven by the broker. All the loop_forever() thread does is handle the inbound messages from the broker (and if needed the response for high QOS messages).
Also if you have a persistent session, then there should be no need to re-subscribe to the topics as the subscription should still be attached to the session in the broker, which will be resumed when the client reconnects (with the same client ID and cleanSession false).
As to which will be delivered first retained or queued mesages, I don't think the spec actually defines which order the broker should send queued messages vs retained messages, which means it may well be broker implementation specific.
Related
I would like read messages from a MQTT broker ordered (chronologically as written) from more than one topic. For example I have topics (which are published to independently by clients at different rates all with QoS 2):
/foo/a
/foo/b
/foo/c
The messages are in a Persistent Session for a long period using Message Expiry Interval and the subscriber could come and go, on and offline, with any number of messages on each topic not yet read.
When I subscribe to: /foo/#, will I receive messages from topics /foo/a interleaved with messages from /foo/b and /foo/c in the order they were received by the broker ?
The specification on Message Ordering says:
... When a Server processes a message that has been published to an Ordered Topic, it MUST send PUBLISH packets to consumers (for the same Topic and QoS) in the order that they were received from any given Client [MQTT-4.6.0-5 ...
"(for the same Topic and QoS)" suggests ordering can only be guaranteed for the same Topic and QoS. So the answer my question of ordering across topics seems to be undefined.. ?
From the Mosquitto broker point of view, if a client has disconnected but has a long session expiry time and has active subscriptions with QoS>0, that is no different to a client that is connected - the session remains open. That means the messages will be delivered according to the ordering requirements in the spec.
This part of the answer covers retained messages only:
My understanding is that message ordering rules only apply for active sessions. That is to say, a client publishes messages and they must be delivered to current consumers only in the same order they were received.
It does not, however, apply to the situation when a client subscribes to a topic filter and receives retained messages. You can get a clue to the intent of the spec there, because the concept of messages being out of order for the same topic and QoS is nonsensical when there is only a single retained message per topic.
Ordering of delivery of retained messages that match a wildcard subscription is undefined. In Mosquitto is it roughly in order of delivery, breadth then depth. This is likely to change in the future to being sorted though.
Is it possible to configure a set of default/hardcoded retained messages to be available on MQTT when Mosquitto starts up?
I would like to use retained messages for app configuration in a microservice architecture to allow modifying and distributing configuration values on runtime. But I need a way of initializing the message when the app is started for the first time.
Another solution would be an application that checks for the existence of the message on startup, publishing the default message if missing, but this is not possible with MQTT since there is no GET or EXISTS operation. A workaround would be to SUBSCRIBE and wait a few seconds before publishing the default message if no callback has been received, which does not seem to be a reliable solution.
As long as you have enabled persistence and a persistence file, in the mosquitto.conf, to store the data in then any retained messages will still be delivered after a broker restart.
As long as you don't clear the retained messages (by publishing a message on the same topic with the retained bit set and a null payload) then they will always be there.
I have a system that relies on a message bus and broker to spread messages and tasks from producers to workers.
It benefits both from being able to do true pub/sub-type communications for the messages.
However, it also needs to communicate tasks. These should be done by a worker and reported back to the broker when/if the worker is finished with the task.
Can MQTT be used to publish this task by a producer, so that it is picked up by a single worker?
In my mind the producer would publish the task with a topic "TASK_FOR_USER_A" and there are X amount of workers subscribed to that topic.
The MQTT broker would then determine that it is a task and send it selectively to one of the workers.
Can this be done or is it outside the scope of MQTT brokers such as Mosquitto?
MQTT v5 has an optional extension called Shared Subscriptions which will deliver messages to a group of subscribers in a round robin approach. So each message will only be delivered to one of the group.
Mosquitto v1.6.x has implemented MQTT v5 and the shared subscription capability.
It's not clear what you mean by 1 message at a time. Messages will be delivered as they arrive and the broker will not wait for one subscriber to finish working on a message before delivering the next message to the next subscriber in the group.
If you have low enough control over the client then you can prevent the high QOS responses to prevent the client from acknowledging the message and force the broker to only allow 1 message to be in flight at a time which would effectively throttle message delivery, but you should only do this if message processing is very quick to prevent the broker from deciding delivery has failed and attempting to deliver the message to another client in the shared group.
Normally the broker will not do any routing above and beyond that based on the topic. The as mentioned in a comment on this answer the Flespi has implemented "sticky sessions" so that messages from a specific publisher will be delivered to the same client in the shared subscription pool, but this is a custom add on and not part of the spec.
What you're looking for is a message broker for a producer/consumer scenario. MQTT is a lightweight messaging protocol which is based on pub/sub model. If you start using any MQTT broker for this, you might face issues depending upon your use case. A few issues to list:
You need ordering of the messages (consumer must get the messages in the same order the producer published those). While QoS 2 guarantees message order without having shared subscriptions, having shared subscriptions doesn't provide ordered topic guarantees.
Consumer gets the message but fails before processing it and the MQTT broker has already acknowledged the message delivery. In this case, the consumer needs to specifically handle the reprocessing of failed messages.
If you go with a single topic with multiple subscribers, you must have idempotency in your consumer.
I would suggest to go for a message broker suitable for this purpose, e.g. Kafka, RabbitMQ to name a few.
As far as I know, MQTT is not meant for this purpose. It doesn't have any internal working to distribute the tasks on workers (consumers). On the Otherhand, AMQP can be used here. One hack would be to conditionalize the workers to accept only a particular type of tasks, but that needs producers to send task type as well. In this case, you won't be able to scale as well.
It's better if you explore other protocols for this type of usecase.
I would like some feedback on this problem and my proposed solution to catching up after missed MQTT messages please:
[Update 1] Simplified problem diagram and added solution diagram. Added mention of QoS
Scenario:
Client A publishes messages that we wish Client B to receive, even if connections are temporarily dropped then restored.
Config
Client A: connect with clean=false. Publish stateful messages with retain = true, non-stateful messages published with retain = false
Client B: connect with clean=false
What will happen
Each time Client A publishes to topic "foo", previous messages are replaced on the broker. Ex. Client A publishes 111, 222, 333. Client B connects after the messages are published. Client B will receive only, 333. Thus, messages 111 and 222 were missed because each message replaced the previous one on that same topic (different topics do not replace each other).
Proposed solution
I envision two types of messages. Stateful and non-stateful. Stateful messages would be things like, voltage, temperature, gps location, pressure. Non-stateful messages would be things like a chat message where history is more likely to be important for context. Missed stateful messages are more likely to be tolerable while non-stateful messages might not be tolerable.
All messages are published with QoS 1 in my case.
For the stateful messages I am thinking Client A will publish with retain = true.
For the non-stateful messages, I am thinking Client A will publish with retain = false (because what good is the last message if we don't have the full historical context of previous messages). When Client B connects/reconnects, I will publish a catch-up (arbitrary name) message containing all the ids of the messages it received, which when Client A receives it, will respond by publishing the whole history of messages minus those in the id list (ids maintained in Client A db). This might work for me if the total aggregate message history isn't too big.
The alternative might be for Client B to send read receipts for each message received.
For me, these two solutions will require a database of messages and some custom logic
This is a follow-up question to this one which I tried answering but was asked to instead form it as an independent, follow-up question.
This is newbie question I'm sure, but using Mosquitto how do I subscribe to a retained message and guarantee that I only get the message once? I have a Paho based java subscriber, and in testing what I notice is, if my subscriber is brought down, and then restarted it pulls some of the same messages that it received before it came down. (Normally it would stay up forever, but stuff happens.) And it's critical the subscriber get each message only once. (For my use case there is only 1 subscriber.)
Is there something built into Mosquitto that provides this mechanism or is this something I have to build into my data schema and retrieval process?
I see there is a way of deleting a message by topic (https://lists.launchpad.net/mosquitto-users/msg00067.html) but right now my topics are pretty generic [Company]/[MAC address]/[Topic x] and it's possible to have two separate messages on the queue with the same topic, and I'm assuming if I send an empty message to a given topic I'll end up deleting all messages of said topic.
Thanks in advance for any help on this.
set the QoS level as 2.
QoS-level-2:
Exactly-once delivery.
This is the highest level that also incurs most over head interms of control messages and the need for locally storing the messages.
Exactly-once is a combination of at-least-once and at-most-once delivery guarantee.