I am a subscriber application.
I receive messages from Publisher. One message at a time.
Question is: I would like to ensure:
I have received same number of messages from my publisher (and have not dropped anything in between)
I have received each message with completeness and accuracy (with all attributes)
Please advise if this is implicitly covered or do I need to do anything special for this?
You should use use a MQ product which has support for transactions and "only once" delivery. By using that kind of software the second point - the accuracy - is guaranteed as you cannot commit transaction if anything goes bad. The first point - no messages dropped or skipped - is implied. With transactions you just can't drop messages. That will not happen. You can skip them if some other subscriber reads them instead of you. But your picture implies that there is one subscriber to each q so you can sleep well.
If you want to prevent tampering with messages you can use something like IBM MQ Advanced Message Security or build your own app level protocol on top of MQ.
ActiveMQ or IBM MQ would fit your usecase perfectly.
Related
Would the mqtt community consider placing message information in the topic name an anti-pattern?
I have a client that has a vast library written around rabbitmq, and I'm trying to tweak their client and server code to allow them to configure their services for mosquitto instead. One central requirement for them is TTL, the clients can sometimes sit for hours publishing data before the server comes back online and they do not want messages to show up that are beyond their TTL.
Their message envelope system is an elaborate json and 1) it would be painful to wrap or alter this json 2) I do not want to incur the expense of unmarshalling json to retrieve a timestamp.
The easiest thing to do is place the timestamp at the end of the topic and consume with wildcards: mytopic/mysubtopic/{timestamp} consumed by mytopic/mysubtopic/#
Are there any unintended consequences for this, and would this be considered an anti-pattern?
Whether this is an anti-pattern is a matter of opinion; the spec defines the topic as "The label attached to an Application Message..." so does not preclude your usage. I can think of a few potential "unintended consequences" to your approach (which may, or may not, apply to your specific situation):
Retain flag: As per your comment you will not be able to set the Retain flag to 1 (because all messages would be retained).
Latest Message only when comms re-established: A subscriber may only want the latest message when communications are re-established. This can be achieved by publishing messages with the retain flag set to 1 which results in your subscriber receiving the latest message (and only the latest message; subject to QOS/CleanSession) on each topic it subscribes to (docs). As per the above this will not work with your topic structure.
Order of delivery: the spec requires that "A Server MUST by default treat each Topic as an 'Ordered Topic'" but there is no such guarantee across topics. Note that ordered delivery is dependent upon settings (see the "Non normative comment" in the spec) so this may not be an issue.
Topic Alias: MQTT V5 introduces Topic Alias which can be used to reduce the amount of data transmitted. This will not provide a benefit with your structure.
I would like some feedback on this problem and my proposed solution to catching up after missed MQTT messages please:
[Update 1] Simplified problem diagram and added solution diagram. Added mention of QoS
Scenario:
Client A publishes messages that we wish Client B to receive, even if connections are temporarily dropped then restored.
Config
Client A: connect with clean=false. Publish stateful messages with retain = true, non-stateful messages published with retain = false
Client B: connect with clean=false
What will happen
Each time Client A publishes to topic "foo", previous messages are replaced on the broker. Ex. Client A publishes 111, 222, 333. Client B connects after the messages are published. Client B will receive only, 333. Thus, messages 111 and 222 were missed because each message replaced the previous one on that same topic (different topics do not replace each other).
Proposed solution
I envision two types of messages. Stateful and non-stateful. Stateful messages would be things like, voltage, temperature, gps location, pressure. Non-stateful messages would be things like a chat message where history is more likely to be important for context. Missed stateful messages are more likely to be tolerable while non-stateful messages might not be tolerable.
All messages are published with QoS 1 in my case.
For the stateful messages I am thinking Client A will publish with retain = true.
For the non-stateful messages, I am thinking Client A will publish with retain = false (because what good is the last message if we don't have the full historical context of previous messages). When Client B connects/reconnects, I will publish a catch-up (arbitrary name) message containing all the ids of the messages it received, which when Client A receives it, will respond by publishing the whole history of messages minus those in the id list (ids maintained in Client A db). This might work for me if the total aggregate message history isn't too big.
The alternative might be for Client B to send read receipts for each message received.
For me, these two solutions will require a database of messages and some custom logic
This is a follow-up question to this one which I tried answering but was asked to instead form it as an independent, follow-up question.
I want to display delivered and read receipts to users in my messaging platform. I am using Eclipse's Paho library with Mosquitto as the broker. Since Mosquitto does not store messages, which is the best way/plugin to
Display delivered receipts - how to use QoS2 acknowledgement receipts to do this?
Display read receipts - suggest me way to do this
How to store messages so that users can view their chat history? Any architectural insights in mysql will be very helpful.
The quick answers to your questions:
High QOS (1/2) is not end to end delivery confirmation, it is only confirmation between the broker and a client. e.g. a publisher publishing at QOS 2 the confirmation is only between the publisher and the broker, not then onward to the subscriber (who may be subscribed at a different QOS anyway). The only way to do this is to send a separate message from the receiving end back to the sender. Also there may be more than one subscriber to any given topic, so you have to think how this would work.
Again, the only way to do this is with a separate message sent when the message is read
You will have to implement this yourself. The only thing that may help is something like the built in support for storing messages in a database present in some brokers (this is not part of the spec, so totally propitiatory to the implementation) e.g. hivemq
Hardlib's answers are 100% on target, but I'll add some thoughts on implementation.
I think a common misunderstanding with MQTT is that it is really a M2M (machine-to-machine) protocol, not a system for exchanging messages between users. That isn't to say you can't use it for messaging (facebook did) but that exists in a layer on top of MQTT. Put another way, MQTT is designed to route messages between machines with little care about the content of those messages. What this means is that user-level niceties (delivery confirmations etc.) aren't really part of it but instead something that you implement on top of MQTT.
So here are some thoughts about how to implement what you propose on top of MQTT:
Consider a situation in which you have two clients (X & Z) which both have access to the same broker (Y). To have client X confirm it has received a message from client Z, simply have client X send a message to a topic (lets say confirmations/z) that client Z is subscribed to. This is trivial to implement in Python or whatever you are writing your application in. (For example, I use that basic procedure to measure round-trip time on my broker.)
However, given that QoS can guarantee that a broker has received the message (and it could be retained or otherwise held for other clients), I would question if this is really necessary unless it is critical that client Z know exactly when client X receives the message.
Depending on your needs there are any number of ways of providing a history for a topic. See the answers here and here for details on MySQL. But if all you need is a local history of a chat or a record of the activity on a few topics, consider simply outputting payloads with timestamps to a text file or JSON. MySQL feels like overkill unless you are dealing with a high volume of messages or need to compose complex queries.
I am using rabbitmq to communicate between microservices written in ruby on rails. Each service subscribes to a topic. All services are scaled and run as multiple instances based on need.
During subscription bunny moves all the messages from the queue into unacked state. This makes other scaled instances to be just idle, since there is no message in ready state.
Is there a way to limit the number of messages a subscription can fetch, so that other instances can take the remaining messages from the queue.
Based on the information you made available, I'm assuming you're using rubybunny. If this assumption is incorrect (there are other ruby clients available for rabbitmq) let me know and/or check the documentation related to your client.
Back to rubybunny, link provided points to necessary information, quoting it:
For cases when multiple consumers share a queue, it is useful to be
able to specify how many messages each consumer can be sent at once
before sending the next acknowledgement.
In AMQP 0.9.1 parlance this is known as QoS or message prefetching.
Prefetching is configured on a per-channel basis.
To configure prefetching use the Bunny::Channel#prefetch method like so:
ch1 = connection1.create_channel
ch1.prefetch(10)
This is newbie question I'm sure, but using Mosquitto how do I subscribe to a retained message and guarantee that I only get the message once? I have a Paho based java subscriber, and in testing what I notice is, if my subscriber is brought down, and then restarted it pulls some of the same messages that it received before it came down. (Normally it would stay up forever, but stuff happens.) And it's critical the subscriber get each message only once. (For my use case there is only 1 subscriber.)
Is there something built into Mosquitto that provides this mechanism or is this something I have to build into my data schema and retrieval process?
I see there is a way of deleting a message by topic (https://lists.launchpad.net/mosquitto-users/msg00067.html) but right now my topics are pretty generic [Company]/[MAC address]/[Topic x] and it's possible to have two separate messages on the queue with the same topic, and I'm assuming if I send an empty message to a given topic I'll end up deleting all messages of said topic.
Thanks in advance for any help on this.
set the QoS level as 2.
QoS-level-2:
Exactly-once delivery.
This is the highest level that also incurs most over head interms of control messages and the need for locally storing the messages.
Exactly-once is a combination of at-least-once and at-most-once delivery guarantee.