I am trying to self-bridge a mosquitto broker. Let me explain the scenario for which I am doing this.
We have 2 parts to our application. One which is concerned with processing data and other is a modbus service that reads data from PLC devices.
The processing part of the application is capable of handling multiple clients. I want to setup a test and a production tenant for the same client. The reason behind this is that this would give our clients the liberty to play around with the test tenant where as the production won't be affected by this.
The test client both sends as well as receives data from the broker. It is important that we don't send the test tenant's data to our modbus service. The topics follow the following format.
company/service/test/+ for the test client and company/service/prod/+ for the prod client.
The modbus service send data to the same broker in the format company/service/prod/+.
Is there a way for me to remap this topic to company/service/test/+ so that both test and prod clients can receive data from the broker.
address 127.0.0.1:41888
topic /company/values/prod/+ in 2
topic /company/values/prod/+ out 2 "" /company/values/test/+
remote_clientid test
remote_username mqttuser
remote_password broker-123
Remember I have a single broker instance and I'm trying to self bridge on this.
In my above configuration, the remapping doesn't happen cause it's not a valid prefix.
Can someone please help me figure out how to approach this?
The problem is the + on the end of the output topic mapping in
topic /company/values/prod/+ out 2 "" /company/values/test/+
You want
topic + out 2 /company/values/prod/ /company/values/test/
This will strip off /company/values/prod/ and replace it with /company/values/test/
You also want to remove the first topic line (topic /company/values/prod/+ in 2 ) as this will lead to an infinity publish loop for any message published to /company/values/prod/+
p.s. starting topics with a leading / while valid in the spec is a REALLY bad idea, as it breaks things when you get round to needing shared subscriptions and injects an unneeded null to the start of every topic.
Related
I have an application when I'm sending MQTT messages to an IoT platform, the IoT platform has their own broker. The problem arose when the broker went down for 2-3 days, with that I lost 2-3 days worth of data.
I was wondering if there was a way to ensure that all data points are stored, and then sent when the broker come back online in order. I've been testing this with Mosquitto, but I can't seem to get it to work.
Is it a matter of using Quality of Service (QoS)? Does this work even the broker is down, or does it need the broker to communicate with? Or do I need to use persistence or retain?
Yes, you are on the right track, it requires QoS and must be used with the other settings together, you can test under the following conditions:
Initialize your MQTT client with clean session flag set to False and a unique client ID;
Here is an example using Paho python library,
mqttc = mqtt.Client("specify_a_unique_client_id", clean_session=False)
Subscribe to a topic with QoS >= 1;
Publish to a topic with QoS >= 1;
NOTICE: you must specify a unique client ID, so that your broker can still recognize the previous client session in case it reconnects. Leave the client ID as empty will auto generate a new one.
Bonus, Here is a good series of articles to explain all the configurations in MQTT, in case you want to understand the details.
I am doing a request/response flow using a MQTT broker and I wondered if brokers like VerneMQ or Mosquitto deal well with huge amount of topics. Basically every time I want to do a request/response, I publish to a topic that looks like rpc/{UUID} meaning every request creates a new topic and then unsubscribe from it when the response is received. Will this come and bite me later ?
Topics are effectively ephemeral already.
Usually the only overhead to a topic is in the list of subscribed topic patterns (because they can be wildcards) held for each client. The topic is read from an incoming messages and checked against this list.
Using UUIDs in topics should not cause any problems.
I'm trying to create a reasonable setup for client-client-communication for our existing infrastructure. I've been reading the docs for Spring, Websocket, STOMP, SockJS and ActiveMQ for over a week now and I'm not not sure whether what I am trying to do is feasible or sensible. Spring server and JavaScript client were up and running relatively quickly and sending messages between clients just works (direct connection from JS client to Spring server). This setup won't suffice for our needs so we decided to put dedicated brokers in between. Configuring ActiveMQ is a nightmare probably because I don't really know where to start. I have not worked with a dedicated broker so far.
Environment
170 independent servers (Tomcat, Spring, SockJS, STOMP)
2 ActiveMQ (Artemis) brokers (load balance, failure safety)
a few thousand clients (JavaScript/.NET, SockJS, STOMP)
Requirement
I need every client to be able to talk to every other client. Every message has to be curated by one of the servers. I'd like the clients to connect to one of the ActiveMQ brokers. The ActiveMQ brokers would hold a single connection to every single server. The point is to avoid that all my clients would have to open 170 WebSocket connections to all the servers. The servers do not need to talk to each other (yet/necessarily) since they are independent with different responsibilities.
Question
Is ActiveMQ or any other dedicated broker viable as transparent proxy/relay i.e. can it handle this situation and are there ways to dynamically decide the correct recipients or should I go another route like rolling my own Spring-based relay?
In a traditional messaging use-case (e.g. using ActiveMQ Artemis with STOMP) the broker manages "destinations" and any messages sent to those destinations. Messages are only dispatched to clients if the client specifically creates a consumer on a destination.
In your use-case all of your 170 "servers" would actually be messaging clients. They would need to create a consumer on the broker in order to receive messages. To be clear, once the consumer is created then the broker would dispatch messages to it as soon as they arrived.
I'm not sure what exactly you mean by "transparent," but if it means that the process(es) receiving the message(s) don't have to do anything then no message broker will handle your use-case.
I implemented a MQTT message broker using mosquitto on my network. I have one web app publishing things to the broker and several servers that subscribed the same topic. So i have a redundancy scenario.
My question is, using mosquitto alone, is there any way to configure it to publish data only on the first subscriber? Otherwise, all of them will do the same thing.
I don't think that is possible.
But you can do this.
Have the first subscriber program respond with an ack on the channel as soon as it gets the message, and have the redundancy program look for the ack for a small time after the initial message.
IF the ack is received the redundancy should not do anything.
So if the first subscriber gets and uses the message, the others wont do anything even if they get the message.
No this is not possible with mosquitto at the moment (without communication between the 2 subscribers as described in the other answer).
For the new release of the MQTT spec (v5)* there is a new mode called "Shared Subscriptions". This allow s multiple clients to subscribe to a single topic and messages will be delivered by round robin to each client. This is more for load balancing rather than master/slave fail over.
*There are some brokers (HiveMQ, IBM MessageSight) that already support some version of Shared Subscriptions at MQTT v3.1.1, but they implement it in slightly different ways (different topic prefixes) so they are not cross compatible.
I want to display delivered and read receipts to users in my messaging platform. I am using Eclipse's Paho library with Mosquitto as the broker. Since Mosquitto does not store messages, which is the best way/plugin to
Display delivered receipts - how to use QoS2 acknowledgement receipts to do this?
Display read receipts - suggest me way to do this
How to store messages so that users can view their chat history? Any architectural insights in mysql will be very helpful.
The quick answers to your questions:
High QOS (1/2) is not end to end delivery confirmation, it is only confirmation between the broker and a client. e.g. a publisher publishing at QOS 2 the confirmation is only between the publisher and the broker, not then onward to the subscriber (who may be subscribed at a different QOS anyway). The only way to do this is to send a separate message from the receiving end back to the sender. Also there may be more than one subscriber to any given topic, so you have to think how this would work.
Again, the only way to do this is with a separate message sent when the message is read
You will have to implement this yourself. The only thing that may help is something like the built in support for storing messages in a database present in some brokers (this is not part of the spec, so totally propitiatory to the implementation) e.g. hivemq
Hardlib's answers are 100% on target, but I'll add some thoughts on implementation.
I think a common misunderstanding with MQTT is that it is really a M2M (machine-to-machine) protocol, not a system for exchanging messages between users. That isn't to say you can't use it for messaging (facebook did) but that exists in a layer on top of MQTT. Put another way, MQTT is designed to route messages between machines with little care about the content of those messages. What this means is that user-level niceties (delivery confirmations etc.) aren't really part of it but instead something that you implement on top of MQTT.
So here are some thoughts about how to implement what you propose on top of MQTT:
Consider a situation in which you have two clients (X & Z) which both have access to the same broker (Y). To have client X confirm it has received a message from client Z, simply have client X send a message to a topic (lets say confirmations/z) that client Z is subscribed to. This is trivial to implement in Python or whatever you are writing your application in. (For example, I use that basic procedure to measure round-trip time on my broker.)
However, given that QoS can guarantee that a broker has received the message (and it could be retained or otherwise held for other clients), I would question if this is really necessary unless it is critical that client Z know exactly when client X receives the message.
Depending on your needs there are any number of ways of providing a history for a topic. See the answers here and here for details on MySQL. But if all you need is a local history of a chat or a record of the activity on a few topics, consider simply outputting payloads with timestamps to a text file or JSON. MySQL feels like overkill unless you are dealing with a high volume of messages or need to compose complex queries.