I'm trying to use ActiveMQ Artemis to receive a single MQTT message, and then clone it to multiple topics.
For example, a single MQTT message with the topic "mydata/PC" is published to ActiveMQ Artemis.
Through some mechanism, ActiveMQ Artemis copies the message, and changes the topic of the first one to "mydata/PC/1" and the second one to "mydata/PC/2".
The listeners (which are Telegraf plug-ins) are listening on PC 1 and the other on PC 2. They would each get their own topic and publish it to their respective InfluxDB databases.
In case PC 2 is down or segmented the listener on that PC will not receive the message (which arrives on PC 1), and therefore the message will be not be cleared from ActiveMQ Artemis until the listener comes back up.
Does this approach sound feasible?
If not, another possibility is if there were some way to specify that the message can't be cleared until it is read twice or n amount of times. That would also work.
This is similar the the JMS concept of a message not being deleted until all subscribers to a topic have been successfully notified. Unfortunately, there are no pre-written telegraph consumers for JMS.
The MQTT protocol itself can already handle this use-case. You don't need to use ActiveMQ Artemis to receive a single MQTT message and then clone it to multiple topics.
Each of your Telegraf plug-ins can create a subscription on "mydata/PC" with CleanSession flag set to false and both of them will receive every message sent to that topic. If they get disconnected they can reconnect and pick up right where they left off (since CleanSession = false will maintain the subscription while the client is disconnected).
Related
I have a system that relies on a message bus and broker to spread messages and tasks from producers to workers.
It benefits both from being able to do true pub/sub-type communications for the messages.
However, it also needs to communicate tasks. These should be done by a worker and reported back to the broker when/if the worker is finished with the task.
Can MQTT be used to publish this task by a producer, so that it is picked up by a single worker?
In my mind the producer would publish the task with a topic "TASK_FOR_USER_A" and there are X amount of workers subscribed to that topic.
The MQTT broker would then determine that it is a task and send it selectively to one of the workers.
Can this be done or is it outside the scope of MQTT brokers such as Mosquitto?
MQTT v5 has an optional extension called Shared Subscriptions which will deliver messages to a group of subscribers in a round robin approach. So each message will only be delivered to one of the group.
Mosquitto v1.6.x has implemented MQTT v5 and the shared subscription capability.
It's not clear what you mean by 1 message at a time. Messages will be delivered as they arrive and the broker will not wait for one subscriber to finish working on a message before delivering the next message to the next subscriber in the group.
If you have low enough control over the client then you can prevent the high QOS responses to prevent the client from acknowledging the message and force the broker to only allow 1 message to be in flight at a time which would effectively throttle message delivery, but you should only do this if message processing is very quick to prevent the broker from deciding delivery has failed and attempting to deliver the message to another client in the shared group.
Normally the broker will not do any routing above and beyond that based on the topic. The as mentioned in a comment on this answer the Flespi has implemented "sticky sessions" so that messages from a specific publisher will be delivered to the same client in the shared subscription pool, but this is a custom add on and not part of the spec.
What you're looking for is a message broker for a producer/consumer scenario. MQTT is a lightweight messaging protocol which is based on pub/sub model. If you start using any MQTT broker for this, you might face issues depending upon your use case. A few issues to list:
You need ordering of the messages (consumer must get the messages in the same order the producer published those). While QoS 2 guarantees message order without having shared subscriptions, having shared subscriptions doesn't provide ordered topic guarantees.
Consumer gets the message but fails before processing it and the MQTT broker has already acknowledged the message delivery. In this case, the consumer needs to specifically handle the reprocessing of failed messages.
If you go with a single topic with multiple subscribers, you must have idempotency in your consumer.
I would suggest to go for a message broker suitable for this purpose, e.g. Kafka, RabbitMQ to name a few.
As far as I know, MQTT is not meant for this purpose. It doesn't have any internal working to distribute the tasks on workers (consumers). On the Otherhand, AMQP can be used here. One hack would be to conditionalize the workers to accept only a particular type of tasks, but that needs producers to send task type as well. In this case, you won't be able to scale as well.
It's better if you explore other protocols for this type of usecase.
I have MQTT Broker (Mosquitto) in different two devices.
They are working. I set bridge between them.
I can publish data from one to another with pub_mosquitto.
I want to get data from any topic in internet broker(broker.hivemq.com) like http://www.mqtt-dashboard.com/
I can take data from any topic in that website brokers but i can not publish taken data to another my broker at the same time.
I want to use my broker pub/sub.
My broker will take data from any broker and publish it to my bridged broker. How can i do this ?
I drew and wrote below:
MQTT bridges can be configured to work in multiple ways
To take local messages and forward them to a remote broker
To take remote messages and forward them to the local broker
To do both of the previous statements
e.g. for mosquitto
topic # out // local to remote
topic # in // remote to local
topic # both // bridge in both directions
Full details on how to configure mosquitto bridges can be found in the man page here
The only limit on what topics can be bridged is based on the normal ACL for the user the local broker uses to authenticate with the remote broker.
To achieve what you want you need to create 2 bridges, one between Broker 1 & Broker 2 that should have the direction of both. This can be setup in the config for either Broker 1 or Broker 2 (but not on both).
Secondly on Broker 1 you can create a bridge to the HiveMQ broker. This should probably have a topic mapping line like:
topic temperature in
This will take any messages published to the HiveMQ broker and bridge them onto Broker 1 which will then bridge them to Broker 2
I have several questions about this topic so I will post them here altogether, otherwise I have to make separate post about separate questions.
The scenario is that I have NodeMCU board which I will be using to receive multiple messages from multiple publishers through a MQTT broker(MQTT box). The following are the queries whose clarification I need, I still haven't implemented yet, I am just forming the logic. Lets say I will be receiving messages from 4 NodeMCU board continuously though MQTT broker, which I will then send to server for storing, the problem is
Will the message of 1 publisher override the message of another publisher before I can store them in server?
The messages that comes from MQTT broker, are they stored in ROM or RAM of NodeMCU board?
Can a MQTT broker send data from 2 publishers concurrently to the same NodeMCU board?
The post is a hot mess, sorry about that.
Quick answers:
If 2 publishers publish on the same topic then messages will be delivered to any subscribers to that topic in the order they are published. So Publisher 1's message will arrive, then Publisher 2's message. Since messages are received sequentially you should not loose a message because it is interrupted by the next message.
Arriving messages will be in RAM when they are delivered
Brokers just deliver messages to any client subscribed to a topic, the client will not even know that there are multiple publishers (there is no publisher specific information in a MQTT message), just that it receives multiple messages on that specific topic
I implemented a MQTT message broker using mosquitto on my network. I have one web app publishing things to the broker and several servers that subscribed the same topic. So i have a redundancy scenario.
My question is, using mosquitto alone, is there any way to configure it to publish data only on the first subscriber? Otherwise, all of them will do the same thing.
I don't think that is possible.
But you can do this.
Have the first subscriber program respond with an ack on the channel as soon as it gets the message, and have the redundancy program look for the ack for a small time after the initial message.
IF the ack is received the redundancy should not do anything.
So if the first subscriber gets and uses the message, the others wont do anything even if they get the message.
No this is not possible with mosquitto at the moment (without communication between the 2 subscribers as described in the other answer).
For the new release of the MQTT spec (v5)* there is a new mode called "Shared Subscriptions". This allow s multiple clients to subscribe to a single topic and messages will be delivered by round robin to each client. This is more for load balancing rather than master/slave fail over.
*There are some brokers (HiveMQ, IBM MessageSight) that already support some version of Shared Subscriptions at MQTT v3.1.1, but they implement it in slightly different ways (different topic prefixes) so they are not cross compatible.
With an MQTT broker, is it possible to set up multiple consumers for a topic such that for any given message on that topic only one consumer will receive the message?
The short answer is no, not with any broker that purely implements the MQTT spec.
I suppose it would be possible to write a broker that talked to the clients using MQTT and only delivered messages to a single subscriber. (It would have to deliver with QOS2 to ensure that every message was consumed)
By coincidence I was talking to a colleague about something similar to this sort of thing earlier in the week, he had found a way to do it using IBM* MQ Light and something called 'Shared Destinations'. (MQ Light uses AMPQ not MQTT)
https://developer.ibm.com/messaging/mq-light/
full disclosure, I work for IBM
UPDATE:
I've since been informed that the IBM MessageSight v1.2 appliance can actually do shared destinations using MQTT (http://www-03.ibm.com/software/products/en/messagesight)
UPDATE 2:
Shared subscriptions is an optional part of the MQTT v5 spec so worth checking any v5 brokers for the option.
Look at Shared Subscriptions https://issues.oasis-open.org/browse/MQTT-234
some MQTT servers support it.
EMQTT (open source):
https://github.com/emqtt/emqttd/issues/639#issuecomment-247851593
HiveMQ:
http://www.hivemq.com/blog/mqtt-client-load-balancing-with-shared-subscriptions/
IBM MessageSight:
http://www.ibm.com/support/knowledgecenter/SSCGGQ_1.2.0/com.ibm.ism.doc/Developing/devsharedsubscriptions.html
VerneMQ:
https://vernemq.com/docs/configuration/balancing.html
That is not possible. In MQTT all subscribers to a particular topic receive messages published to said topic. In order to direct a message to a particular subscriber, both publisher and subscriber would have to use a particular topic different to that used by other subscribers.
Independent of the broker that you're are using, you can use Apache Camel to implement a route that copies all messages from Topic A to Topic B.
Or copy only specific messages that match an specific rule such as user, content pattern, QoS.
Other solution is using a multi-protocol broker such as ActiveMQ and copy specific message topics to a Queue (queues only can have one consumer) and consume the queue with another protocol such as JMS or STOMP.