I'm trying to help a customer connect their Mosquitto bridge to Azure IoT Edge. They have some legacy equipment that speaks MQTT, but because it can't do TLS and the topics can't be changed, we are trying to run the messages through the Mosquitto MQTT Broker, and over to IoT Edge via the Mosquitto bridge...
I've had no problems getting the actual connection made from the bridge to IoT Edge and I have messages flowing to the bridge. That connectivity works fine. The problem comes in the topics. I really can't change the topic structure that the client publish on. However, IoT Edge requires messages to be published on a specific MQTT topic (devices//messages/events). Where device_id is the name of my broker, let's say 'mymqttbroker' just for fun.
So, what I'm trying to do is to take the messages that some in on pretty much any topic, and resend those messages through the bridge on the devices/mymqttbroker/messages/events topic to IoT Edge.
I know the topic line in the bridge config has the remote_prefix and local_prefix parameters, but that won't cut it. Per this article, it says you can't do this..
"E.g A broker would receive messages to topic sensor1 and remap them to new_sensor1. Currently this form of remapping is not available,"
Any idea how to do something like this? is it possible? Essentially, is there any way in the bridge to accept messages from any topic, and republish them on a specific fixed topic?
The quick and dirty way is to write a little helper app that subscribes to the old topics and republishes to the new topics, then just bridge the new topics.
It does add another point of failure, but it's the only option for mosquitto.
If you are not wedded to mosquitto, you can build your own custom broker with something link mosca and add the remapping into the broker.
Related
I have the doubt if a MQTT Broker also can be a client? Or I must need to separate and get a dispositive to act as a broker and another as a client. I'm not finding information on internet.
If I understand the question properly, what you are looking for is MQTT bridging.
This is where one broker acts as a client to a second (or multiple) broker and based on config copies messages on topics between the 2 brokers.
How this is configured depends on which broker you are using, but the concept is part of the spec (I don't think it's optional off the top of my head) so all brokers should support it
I am willing to create a mediator which is subscribed and published to 2 separate broker who have no access to topics of each other. The aim is to updates and create a logic of the message published by broker 1 and send it to broker 2 according to the set of rules
Do I need 2 separate ports ? As the topic level might be different in both brokers
Any help is much appreciated!!!
There is no MQTT standard (note, can only speak for 3.1.1) defined features that would allow a client maintain two concurrent connections. Therefore, this is entirely broker implementation-dependent and necessitates a bridge.
For example, the Eclipse Mosquitto broker can be configured as a bridge to another broker and even remap topics from itself to a different topic structure of the other. Please refer to the Mosquitto man page section Configuring Bridges for the specifics.
As far as creating a bespoke application, you can always write a simple Python program that is running two instances of an MQTT client (Eclipse Paho for example, which has a lightweight asyncio wrapper to facilitate concurrency), each connected to different brokers. The glue logic between them just has to re-publish an incoming subscribed topic message from Broker A to some topic, with or without a remapping step, to Broker B.
If the two brokers are both running locally on a single NIC, then you would need to use different ports.
I implemented a MQTT message broker using mosquitto on my network. I have one web app publishing things to the broker and several servers that subscribed the same topic. So i have a redundancy scenario.
My question is, using mosquitto alone, is there any way to configure it to publish data only on the first subscriber? Otherwise, all of them will do the same thing.
I don't think that is possible.
But you can do this.
Have the first subscriber program respond with an ack on the channel as soon as it gets the message, and have the redundancy program look for the ack for a small time after the initial message.
IF the ack is received the redundancy should not do anything.
So if the first subscriber gets and uses the message, the others wont do anything even if they get the message.
No this is not possible with mosquitto at the moment (without communication between the 2 subscribers as described in the other answer).
For the new release of the MQTT spec (v5)* there is a new mode called "Shared Subscriptions". This allow s multiple clients to subscribe to a single topic and messages will be delivered by round robin to each client. This is more for load balancing rather than master/slave fail over.
*There are some brokers (HiveMQ, IBM MessageSight) that already support some version of Shared Subscriptions at MQTT v3.1.1, but they implement it in slightly different ways (different topic prefixes) so they are not cross compatible.
I have started reading some details about MQTT protocol and its implementation. I came across the term 'cluster' a lot. Can anyone help me understand what does 'cluster' mean for MQTT protocol?
In this comparison of various MQTT protocol, there is a column for the term 'cluster'
Forwarding messages with topic bridge loops will not result in a true MQTT broker cluster, which will lead to drawbacks lined out above.
A true MQTT broker cluster is a distributed system that represents one logical MQTT broker. A cluster consists of various individual MQTT broker nodes, that are typically installed on separate physical or virtual machines and or connect over a network.
Typical advantages of MQTT broker clusters include:
Elimination of the single point of failure
Load distribution across multiple cluster nodes
The ability for clients to resume sessions on any broker cluster
Scalability
Resilience and fault tolerance - especially useful in cloud environments
I recommend this blogpost, if you're looking for a more detailed explanation.
A cluster is a collection of MQTT brokers set up to bridge all topics between each other so that a client can connect to any one of the cluster members and still publish and receive messages to all other clients no matter which cluster member they are connected to.
A few things to be aware of:
Topic bridge loops, where a message is published to one cluster member which is then forwarded to another cluster member, then another and finally back to the original. If this happens the original broker doesn't have a way to know it originally pushed this to the other cluster members so the message and end up in a loop. Shared message state databases or using a single bridging replication broker can fix this.
Persistent subscriptions/sessions, unless brokers have a pooled session cache then clients will not retain session or subscription status if they connect to a different cluster member when reconnecting.
With an MQTT broker, is it possible to set up multiple consumers for a topic such that for any given message on that topic only one consumer will receive the message?
The short answer is no, not with any broker that purely implements the MQTT spec.
I suppose it would be possible to write a broker that talked to the clients using MQTT and only delivered messages to a single subscriber. (It would have to deliver with QOS2 to ensure that every message was consumed)
By coincidence I was talking to a colleague about something similar to this sort of thing earlier in the week, he had found a way to do it using IBM* MQ Light and something called 'Shared Destinations'. (MQ Light uses AMPQ not MQTT)
https://developer.ibm.com/messaging/mq-light/
full disclosure, I work for IBM
UPDATE:
I've since been informed that the IBM MessageSight v1.2 appliance can actually do shared destinations using MQTT (http://www-03.ibm.com/software/products/en/messagesight)
UPDATE 2:
Shared subscriptions is an optional part of the MQTT v5 spec so worth checking any v5 brokers for the option.
Look at Shared Subscriptions https://issues.oasis-open.org/browse/MQTT-234
some MQTT servers support it.
EMQTT (open source):
https://github.com/emqtt/emqttd/issues/639#issuecomment-247851593
HiveMQ:
http://www.hivemq.com/blog/mqtt-client-load-balancing-with-shared-subscriptions/
IBM MessageSight:
http://www.ibm.com/support/knowledgecenter/SSCGGQ_1.2.0/com.ibm.ism.doc/Developing/devsharedsubscriptions.html
VerneMQ:
https://vernemq.com/docs/configuration/balancing.html
That is not possible. In MQTT all subscribers to a particular topic receive messages published to said topic. In order to direct a message to a particular subscriber, both publisher and subscriber would have to use a particular topic different to that used by other subscribers.
Independent of the broker that you're are using, you can use Apache Camel to implement a route that copies all messages from Topic A to Topic B.
Or copy only specific messages that match an specific rule such as user, content pattern, QoS.
Other solution is using a multi-protocol broker such as ActiveMQ and copy specific message topics to a Queue (queues only can have one consumer) and consume the queue with another protocol such as JMS or STOMP.