I am using the mosquitto MQTT Broker.
Also, I have multiple (currently 10, but the number will increase) clients that publish some sensor data periodically to topic A. These clients are technically identical, but do have a unique identifier (serial number).
I also have a client that subscribes topic A in order to receive the published messages and persist the sensor valus in a database.
I certainly need to know which Sensor (i.e. client) has sent a particular value.
As a solution, one could just append some Sensor ID to the payload of each published message. But since the sensors access the broker via GSM, I need to keep the traffic low, so I am trying to avoid that.
I assume, the Broker itself knows which message comes from which client, especially when using perisistent connections, i.e. clean_session=False. Is that correct?
If yes, is there any chance that the subscribing clients can obtain the client_id when receiving the message?
Can it be configured in mosquitto? Or is it default behavior and I am missing something?
I am using paho-mqtt 1.3.1 for all clients.
No, the client id is not part of a published message. It is only used to identify the client to the broker when the connection is established in order to determine if stored messages and persistent subscriptions should be honoured.
The easiest solution is to use a separate topic for each sensor but with a shared root. e.g.
sensor 1 publishes to A/1
sensor 2 published to A/2
The client would then subscribe to A/+ this would then receive all the messages and can use the second half of the topic to determine which sensor it came from.
The other options is as you suggested which is to include the id in the payload.
Sending the client-id with payload(message) is possible. But you need to use delimiters in payload(message) at publisher side . Example: Publisher sends the payload as "client-ID=3 - temperature = 29 " . At the subscriber side , you remove the delimiters using strtok() .
There is no configuration available at broker side.
Per my experience with mosquitto, I don't think there is an option for mosquitto to change either the topic or the payload when re-publishing a received message.
However, I think it is just an implementation issue.
Theoretically, I think it is OK and good to support such kind of feature, since it does not violate MQTT specification at all.
(http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/csprd02/mqtt-v3.1.1-csprd02.html#_Toc385349773, Section 3.3.2.1)
However, since the Server is permitted to override the Topic Name,
it might not be the same as the Topic Name in the original PUBLISH Packet.
The pratical solution for your current problem is, as pointed by #hardillb, either publishing using different topics but receiving using a topic with wildcard (+ or #), or, containing publisher information in payload.
Related
When subscribing to a wildcard and a topic that overlaps that wildcard, the client receives twice the messages because of this overlapping behavior.
Example:
Client-A:
subscribe{aaa/bbb/+/ddd, aaa/bbb/ccc/ddd}
Then, if I publish something on the wildcard topic, Client-A will receive only one:
Client-B:
Publish{aaa/bbb/www/ddd, "blablabla"}
Client-A:
Received{aaa/bbb/www/ddd, "blablabla"}
But, if I publish on the specific topic, Client-A receives twice:
Client-B:
Publish{aaa/bbb/ccc/ddd, "blablabla"}
Client-A:
Received{aaa/bbb/ccc/ddd, "blablabla"}
Received{aaa/bbb/ccc/ddd, "blablabla"}
Checking the MQTT specification, it states this (source, page 36, section 3.3.5 Actions):
When Clients make subscriptions with Topic Filters that include wildcards, it is possible for a Client’s subscriptions to overlap so that a published message might match multiple filters. In this case the Server MUST deliver the message to the Client respecting the maximum QoS of all the matching subscriptions [MQTT-3.3.5-1]. In addition, the Server MAY deliver further copies of the message, one for each additional matching subscription and respecting the subscription’s QoS in each case.
If I use the Paho MQTT-SN Gateway (Server), then the server sends only once. But if I use the Eclipse Public Server, then I receive twice. The behavior is not consistent, or, at least, I cannot expect which behavior will be until I test it.
The question is: is it possible to set the specific behavior for the wildcard overlap to send only once? Or it depends on server-to-server implementation?
As the specification says that either behaviour is correct it becomes a choice for the broker implementers and likewise adding a configuration option to change the behaviour one way or the other would be a choice.
You will have to test any broker you are using and then consult their documentation to see if it is possible to change the behaviour.
There is no generic answer to this question.
I'm trying to run an MQTT broker and I want to store the published data, but I need to know which user sent the message so I can store payload for each user and study them later. The problem is when two different user try to publish message on same topic I can not tell whose data it is. Is there a way to figure out the publisher of a message? I'm using Mosquitto btw.
Short answer, you don't.
MQTT messages do not contain any information about the user or client that sent it, unless you choose to encode it in the message (as part of the payload for v3.x or alternatively in the header properties for v5.0)
Longer answer:
Some MQTT brokers have plugin APIs that may allow you access to more meta data for a message. You may be able to write a plugin that will take the message + the meta data and then store them. Last time I looked, mosquitto's plugin API was only for writing authentication plugins, and did not give access to the messages themselves. But a different broker may allow this.
As far as I can see, there is no packet field where we can derive from which mqtt client lib (eg paho) and version is used to connect to the mqtt broker, right?
I looked in this document and couln't find anything related: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718018
Yes, there is no information about the client at all in the message unless you choose to include it in the payload.
Part of the point of MQTT was to make the messages as small as possible. The only information in a message* is the following:
Topic
QOS Level
Retained flag
Payload
*at least for MQTTv3.x
I made an instant messaging app using MQTT protocol.
I want to add some extra data about messages in payload like sent time ( server time not client time ) and also provide kind of server side payload sanitizing.
Is it a good idea to add a third party client with superuser privileges between message sender and message receiver on broker's local machine to do this job ?
or is there any better idea ?
by the way I'm using EMQTT as message broker.
From a pure security view having direct peer to peer traffic (without filtering and sanitising) sounds like a dangerous idea. (At least in the Internet-of-things domain I would clearly object against it.)
Why? Because the clients are outside of your control (i.e. a hacker can re-engineer) and inject any traffic to exploit security holes on the receiving side of other clients.
So sanitising on the server side sounds like a very good idea.
I would suggest two topics: One (inbound) topic the clients use to publish messages, and another (outbound) topic used by clients to subscribe to messages. A server side component would then read the messages from inbound topic, sanitize it and publish to the outbound topic.
This de-coupeling makes it also easier to introduce MQTT payload changed: If you update the payload in a non-compatible way, introduce a new inbound topic and keep the old inbound topic too. This allows you to support old and new clients during the transition phase.
I am using mosquitto mqtt client.
For example, you have users that publish and subscribe to topics. The topic actually correlates to a REsT endpoint.
Scenario 1 (typical pub/sub usage)
UserA subscribes to topic /device/123/meta
UserB publishes some data to topic /device/123/meta
by definition, this publish is broadcasted out to the subscribers
we have a script subscribed to /devices/# which knows how to save the payload for the topic /device/123/meta when it receives publish'ed data. This data is saved to the database.
Scenario 2
Someone updates data /device/123/meta via a ReST interface (or a direct DB update, the key is it's not a MQTT publish).
database is updated
a publish message is sent to the MQTT broker so that all subscribers get the updates as a payload
Scenario 2 is what I'm trying to wrap my head around. This creates a nasty feedback loop. When internal messages are broadcasted out, my script to deal with publish events from users can't differentiate between publish events originating from a 3rd party user or an internal publish event only meant to broadcast out some data (with no saving of data needed).
How should I handle this? The MQTT message is very simplistic and I'm not finding anything I can base logic off of. I'm trying to explore using the origin somehow, but no luck this far. I realize I can write plugins, but this is quite the task for mosquitto.
There is no way to distinguish where a message originated from the subscriber at a pure MQTT protocol level. Part of the point of a pub/sub protocol is to decouple publishers from subscribers.
The most portable way to do this would be to add a flag to the actual message payload to indicate that message originated from somewhere other than the actual device.
Or assuming the message is being published as a result a trigger in the database have the trigger check if the incoming message actually changed the database stored value, if the incoming messages matches the existing state of the DB then there is no need to republish it.
Mosquitto's plugin mechanism is currently only for writing authentication and authorisation solutions, but the JavaScript mosca or Java HiveMQ brokers support plugins that may be able to do what you want.