What mechanism(s) does Redis use to keep messages in memory in case of pub-sub ? If no client is subscribed what happens to the messages? Will Redis buffer them ? Is there a way to configure the min. and max. memory allocated per channel ?
Redis does not keep messages in memory in the Pub/Sub context as you can see in the implementation (x):
the message is sent to clients listening for that channel (if any),
the message is sent to clients listening to matching channels (if any).
Then Redis simply returns how many clients have received the message (keeping in mind that a client may receive a single message multiple times e.g if multiple pattern matches).
If there is no client subscribed, Redis simply returns 0 and the message is not recorded/buffered:
> publish foo test
(integer) 0
(x) basically Redis loops over the list of subscribed clients and sends a reply with the message.
Related
Assuming I have topic A and topic B.
Upon starting my application, inside on_connect, I am subscribing to both topics with QOS 2. Then I start the thread with loop_forever().
Now, lets assume I have missed messages on topic B which were sent while I was gone and I have retained messages on topic A.
Upon subscribing, which messages will be processed first?
Is it defined due to the type (i.e. persistent session messages and retained messages on a topic)?
Or is the subscription order in on_connect decisive? (which I would not assume as the thread starts checking them upon loop_forever() only and the order might not be presumed)
Or is it random?
From my tests, it seems the missed messages I get due to the persistent session from topic B will be processed first.
Can I rely on that behaviour?
PS: As hardillb mentioned that behaviour might be broker specific. I am using Mosquitto.
First your model is a little wrong, the client doesn't check for anything, message delivery is entirely driven by the broker. All the loop_forever() thread does is handle the inbound messages from the broker (and if needed the response for high QOS messages).
Also if you have a persistent session, then there should be no need to re-subscribe to the topics as the subscription should still be attached to the session in the broker, which will be resumed when the client reconnects (with the same client ID and cleanSession false).
As to which will be delivered first retained or queued mesages, I don't think the spec actually defines which order the broker should send queued messages vs retained messages, which means it may well be broker implementation specific.
I'm running mosquttio as an MQTT broker and I have multiple devices sending sensory data periodically. I want to collect all the messages and store them. My question is, is there any advantage to having multiple connections (each connection has a unique id and subscribed to subset of the topics.) to the broker or is it preferable to have a single connection gathering all the data.
Note: the subscribers will be on the same machine as the broker.
It probably depends on the messages in question and what processing the client is going to do on those messages.
Having a single client subscribed to '#' will mean that there will only be a single entry on the subscribed topic to search for matches when processing a message. But this is probably a negligible amount of overhead under most situations.
If the message rate is high enough and there is any overhead to the storage then using something called Shared Subscriptions to allow a pool of client to all subscribe to the same topic (or wildcard) and ensure that any message is only delivered to a single client in the pool. This means that the processing of messages can be load balanced over the pool of clients.
Using Shared Subscriptions means that you can dynamically add or remove clients from the pool without having to repartition the topic space across the clients.
I have a system that relies on a message bus and broker to spread messages and tasks from producers to workers.
It benefits both from being able to do true pub/sub-type communications for the messages.
However, it also needs to communicate tasks. These should be done by a worker and reported back to the broker when/if the worker is finished with the task.
Can MQTT be used to publish this task by a producer, so that it is picked up by a single worker?
In my mind the producer would publish the task with a topic "TASK_FOR_USER_A" and there are X amount of workers subscribed to that topic.
The MQTT broker would then determine that it is a task and send it selectively to one of the workers.
Can this be done or is it outside the scope of MQTT brokers such as Mosquitto?
MQTT v5 has an optional extension called Shared Subscriptions which will deliver messages to a group of subscribers in a round robin approach. So each message will only be delivered to one of the group.
Mosquitto v1.6.x has implemented MQTT v5 and the shared subscription capability.
It's not clear what you mean by 1 message at a time. Messages will be delivered as they arrive and the broker will not wait for one subscriber to finish working on a message before delivering the next message to the next subscriber in the group.
If you have low enough control over the client then you can prevent the high QOS responses to prevent the client from acknowledging the message and force the broker to only allow 1 message to be in flight at a time which would effectively throttle message delivery, but you should only do this if message processing is very quick to prevent the broker from deciding delivery has failed and attempting to deliver the message to another client in the shared group.
Normally the broker will not do any routing above and beyond that based on the topic. The as mentioned in a comment on this answer the Flespi has implemented "sticky sessions" so that messages from a specific publisher will be delivered to the same client in the shared subscription pool, but this is a custom add on and not part of the spec.
What you're looking for is a message broker for a producer/consumer scenario. MQTT is a lightweight messaging protocol which is based on pub/sub model. If you start using any MQTT broker for this, you might face issues depending upon your use case. A few issues to list:
You need ordering of the messages (consumer must get the messages in the same order the producer published those). While QoS 2 guarantees message order without having shared subscriptions, having shared subscriptions doesn't provide ordered topic guarantees.
Consumer gets the message but fails before processing it and the MQTT broker has already acknowledged the message delivery. In this case, the consumer needs to specifically handle the reprocessing of failed messages.
If you go with a single topic with multiple subscribers, you must have idempotency in your consumer.
I would suggest to go for a message broker suitable for this purpose, e.g. Kafka, RabbitMQ to name a few.
As far as I know, MQTT is not meant for this purpose. It doesn't have any internal working to distribute the tasks on workers (consumers). On the Otherhand, AMQP can be used here. One hack would be to conditionalize the workers to accept only a particular type of tasks, but that needs producers to send task type as well. In this case, you won't be able to scale as well.
It's better if you explore other protocols for this type of usecase.
I have following usage pattern in an application:
The publisher sends messages to a topic with rate 5 microseconds per message (i.e. send one message every 5 micros).
Consumer subscribes to the topic and handles messages with rate 10 microseconds per message (i.e. it takes 10 micros to complete onReceive callback in JAVA API).
The consumer is interested only in the last message published to the topic, so all intermediate not handled messages can be dropped.
Is it possible to avoid the congestion in the queue of unprocessed messages on the consumer side?
I tried to use eliding with delay=0 (documentation link), however, it doesn’t help if the message already put to the internal queue in the Solace on the consumer side (I refer to com.solacesystems.jcsmp.impl.XMLMessageQueue).
Setting delay to some specific value works fine, but it doesn't scale well, because this number is dynamic and depends on the number of publishers and consumer performance.
A possible solution to this would be to create a LVQ (last value queue) which subscribes to the topic. You create a LVQ by setting the queue quota to 0 MB.
Then have your subscribing application consume messages from the LVQ.
In the appliance you should see the nearly same performance as when sending using direct messages as it will never hit the spool.
How many times per minute does the MQTT client poll the server? Is it a big data traffic or not? I know the size of the packet can be small, but how many times the client ping the broker to make itself "online" in the broker.
If I was not clear please comment this question and I'll try explain better my doubt.
My broker is Mosquitto and the clients are small device (sensors and etc.)
Assuming no data flow (which is of course application dependent), the client will periodically send a PINGREQ message to the broker. This is a 2 byte message and the broker replies with a PINGRESP, also 2 bytes.
The rate at which PINGREQ is sent depends on the keepalive parameter set when you connect. This tells the broker the interval at which it should expect at least one message from the client. In the absence of any other message, the client sends a PINGREQ.
60 seconds is often used as a default value (whether or not this is appropriate for you depends on how quickly you want the client/broker to respond to a hung connection). In the absence of any other messages flowing, maintaining the keepalive guarantee would mean 4 bytes total transferred every minute. This is only the application level data of course, the length of the data on the wire will be bigger.