In this post, I asked if it was possible to keep a message in a queue in ActiveMQ Artemis until all subscribers had received it, and the answer was that this can be done using the clean = false flag.
I haven't tested this yet due to corporate firewall restrictions, but I think the MQTT "clean" parameter may be handled in the Telegraf MQTT listener by the "persistent_session = true" parameter - there is no "clean" parameter available.
But I do wonder how MQTT distinguishes between when a particular subscriber is offline vs. when the subscriber process has simply been terminated? If you terminate the Telegraf process, it wouldn't be able to send the unsubscribe command, so the message would be kept permanently or until some message expiration limit has been met (that might be ok, actually).
So basically the question is, how do you specify the "clean" parameter in the Telegraf MQTT listener?
Different MQTT client implementations will represent protocol semantics in different ways. In this case it appears the Telegraf Input Plugin for MQTT represents the CleanSession flag for the CONNECT control packet as persistent_session.
The documentation says this about persistent_session:
## Persistent session disables clearing of the client session on connection.
## In order for this option to work you must also set client_id to identify
## the client. To receive messages that arrived while the client is offline,
## also set the qos option to 1 or 2 and don't forget to also set the QoS when
## publishing.
So setting persistent_session to true would seem to be equivalent to setting the CleanSession flag to false.
Related
I'm trying to use ActiveMQ Artemis to receive a single MQTT message, and then clone it to multiple topics.
For example, a single MQTT message with the topic "mydata/PC" is published to ActiveMQ Artemis.
Through some mechanism, ActiveMQ Artemis copies the message, and changes the topic of the first one to "mydata/PC/1" and the second one to "mydata/PC/2".
The listeners (which are Telegraf plug-ins) are listening on PC 1 and the other on PC 2. They would each get their own topic and publish it to their respective InfluxDB databases.
In case PC 2 is down or segmented the listener on that PC will not receive the message (which arrives on PC 1), and therefore the message will be not be cleared from ActiveMQ Artemis until the listener comes back up.
Does this approach sound feasible?
If not, another possibility is if there were some way to specify that the message can't be cleared until it is read twice or n amount of times. That would also work.
This is similar the the JMS concept of a message not being deleted until all subscribers to a topic have been successfully notified. Unfortunately, there are no pre-written telegraph consumers for JMS.
The MQTT protocol itself can already handle this use-case. You don't need to use ActiveMQ Artemis to receive a single MQTT message and then clone it to multiple topics.
Each of your Telegraf plug-ins can create a subscription on "mydata/PC" with CleanSession flag set to false and both of them will receive every message sent to that topic. If they get disconnected they can reconnect and pick up right where they left off (since CleanSession = false will maintain the subscription while the client is disconnected).
According to the MQTT specification, a QoS 2 message sent by a MQTT client must follow this workflow:
During the various phases Mosquitto stores the message in its memory. This is also confirmed by looking at the mosquitto.db persistent storage using the db_dump tool described here.
The question is: if a malicious client PUBLISH tons of messages with QoS 2 but never sends the PUBREL message as a response to PUBREC what happens ? Mosquitto keep the messages undefinitively ? I expected some kind of configuration parameter able to get rid of such unacknowledged messages after some time but I can't find any.
I'm not 100% sure, but I think the max_inflight_messages setting should kick in here and not allow the client to send the second QOS 2 message until the first has been completed.
This would limit each client to one malicious message at time.
There are a number of libraries that will allow you low level control over when packets are sent so building a PoC shouldn't be that hard. And if it is possible to trigger a DoS style attack, I'm sure eclipse/mosquitto would look kindly at a Pull Request with a fix.
I have an application when I'm sending MQTT messages to an IoT platform, the IoT platform has their own broker. The problem arose when the broker went down for 2-3 days, with that I lost 2-3 days worth of data.
I was wondering if there was a way to ensure that all data points are stored, and then sent when the broker come back online in order. I've been testing this with Mosquitto, but I can't seem to get it to work.
Is it a matter of using Quality of Service (QoS)? Does this work even the broker is down, or does it need the broker to communicate with? Or do I need to use persistence or retain?
Yes, you are on the right track, it requires QoS and must be used with the other settings together, you can test under the following conditions:
Initialize your MQTT client with clean session flag set to False and a unique client ID;
Here is an example using Paho python library,
mqttc = mqtt.Client("specify_a_unique_client_id", clean_session=False)
Subscribe to a topic with QoS >= 1;
Publish to a topic with QoS >= 1;
NOTICE: you must specify a unique client ID, so that your broker can still recognize the previous client session in case it reconnects. Leave the client ID as empty will auto generate a new one.
Bonus, Here is a good series of articles to explain all the configurations in MQTT, in case you want to understand the details.
I have a system that relies on a message bus and broker to spread messages and tasks from producers to workers.
It benefits both from being able to do true pub/sub-type communications for the messages.
However, it also needs to communicate tasks. These should be done by a worker and reported back to the broker when/if the worker is finished with the task.
Can MQTT be used to publish this task by a producer, so that it is picked up by a single worker?
In my mind the producer would publish the task with a topic "TASK_FOR_USER_A" and there are X amount of workers subscribed to that topic.
The MQTT broker would then determine that it is a task and send it selectively to one of the workers.
Can this be done or is it outside the scope of MQTT brokers such as Mosquitto?
MQTT v5 has an optional extension called Shared Subscriptions which will deliver messages to a group of subscribers in a round robin approach. So each message will only be delivered to one of the group.
Mosquitto v1.6.x has implemented MQTT v5 and the shared subscription capability.
It's not clear what you mean by 1 message at a time. Messages will be delivered as they arrive and the broker will not wait for one subscriber to finish working on a message before delivering the next message to the next subscriber in the group.
If you have low enough control over the client then you can prevent the high QOS responses to prevent the client from acknowledging the message and force the broker to only allow 1 message to be in flight at a time which would effectively throttle message delivery, but you should only do this if message processing is very quick to prevent the broker from deciding delivery has failed and attempting to deliver the message to another client in the shared group.
Normally the broker will not do any routing above and beyond that based on the topic. The as mentioned in a comment on this answer the Flespi has implemented "sticky sessions" so that messages from a specific publisher will be delivered to the same client in the shared subscription pool, but this is a custom add on and not part of the spec.
What you're looking for is a message broker for a producer/consumer scenario. MQTT is a lightweight messaging protocol which is based on pub/sub model. If you start using any MQTT broker for this, you might face issues depending upon your use case. A few issues to list:
You need ordering of the messages (consumer must get the messages in the same order the producer published those). While QoS 2 guarantees message order without having shared subscriptions, having shared subscriptions doesn't provide ordered topic guarantees.
Consumer gets the message but fails before processing it and the MQTT broker has already acknowledged the message delivery. In this case, the consumer needs to specifically handle the reprocessing of failed messages.
If you go with a single topic with multiple subscribers, you must have idempotency in your consumer.
I would suggest to go for a message broker suitable for this purpose, e.g. Kafka, RabbitMQ to name a few.
As far as I know, MQTT is not meant for this purpose. It doesn't have any internal working to distribute the tasks on workers (consumers). On the Otherhand, AMQP can be used here. One hack would be to conditionalize the workers to accept only a particular type of tasks, but that needs producers to send task type as well. In this case, you won't be able to scale as well.
It's better if you explore other protocols for this type of usecase.
I implemented a MQTT message broker using mosquitto on my network. I have one web app publishing things to the broker and several servers that subscribed the same topic. So i have a redundancy scenario.
My question is, using mosquitto alone, is there any way to configure it to publish data only on the first subscriber? Otherwise, all of them will do the same thing.
I don't think that is possible.
But you can do this.
Have the first subscriber program respond with an ack on the channel as soon as it gets the message, and have the redundancy program look for the ack for a small time after the initial message.
IF the ack is received the redundancy should not do anything.
So if the first subscriber gets and uses the message, the others wont do anything even if they get the message.
No this is not possible with mosquitto at the moment (without communication between the 2 subscribers as described in the other answer).
For the new release of the MQTT spec (v5)* there is a new mode called "Shared Subscriptions". This allow s multiple clients to subscribe to a single topic and messages will be delivered by round robin to each client. This is more for load balancing rather than master/slave fail over.
*There are some brokers (HiveMQ, IBM MessageSight) that already support some version of Shared Subscriptions at MQTT v3.1.1, but they implement it in slightly different ways (different topic prefixes) so they are not cross compatible.