Is it possible to configure a set of default/hardcoded retained messages to be available on MQTT when Mosquitto starts up?
I would like to use retained messages for app configuration in a microservice architecture to allow modifying and distributing configuration values on runtime. But I need a way of initializing the message when the app is started for the first time.
Another solution would be an application that checks for the existence of the message on startup, publishing the default message if missing, but this is not possible with MQTT since there is no GET or EXISTS operation. A workaround would be to SUBSCRIBE and wait a few seconds before publishing the default message if no callback has been received, which does not seem to be a reliable solution.
As long as you have enabled persistence and a persistence file, in the mosquitto.conf, to store the data in then any retained messages will still be delivered after a broker restart.
As long as you don't clear the retained messages (by publishing a message on the same topic with the retained bit set and a null payload) then they will always be there.
Related
Assuming I have topic A and topic B.
Upon starting my application, inside on_connect, I am subscribing to both topics with QOS 2. Then I start the thread with loop_forever().
Now, lets assume I have missed messages on topic B which were sent while I was gone and I have retained messages on topic A.
Upon subscribing, which messages will be processed first?
Is it defined due to the type (i.e. persistent session messages and retained messages on a topic)?
Or is the subscription order in on_connect decisive? (which I would not assume as the thread starts checking them upon loop_forever() only and the order might not be presumed)
Or is it random?
From my tests, it seems the missed messages I get due to the persistent session from topic B will be processed first.
Can I rely on that behaviour?
PS: As hardillb mentioned that behaviour might be broker specific. I am using Mosquitto.
First your model is a little wrong, the client doesn't check for anything, message delivery is entirely driven by the broker. All the loop_forever() thread does is handle the inbound messages from the broker (and if needed the response for high QOS messages).
Also if you have a persistent session, then there should be no need to re-subscribe to the topics as the subscription should still be attached to the session in the broker, which will be resumed when the client reconnects (with the same client ID and cleanSession false).
As to which will be delivered first retained or queued mesages, I don't think the spec actually defines which order the broker should send queued messages vs retained messages, which means it may well be broker implementation specific.
I implemented a MQTT message broker using mosquitto on my network. I have one web app publishing things to the broker and several servers that subscribed the same topic. So i have a redundancy scenario.
My question is, using mosquitto alone, is there any way to configure it to publish data only on the first subscriber? Otherwise, all of them will do the same thing.
I don't think that is possible.
But you can do this.
Have the first subscriber program respond with an ack on the channel as soon as it gets the message, and have the redundancy program look for the ack for a small time after the initial message.
IF the ack is received the redundancy should not do anything.
So if the first subscriber gets and uses the message, the others wont do anything even if they get the message.
No this is not possible with mosquitto at the moment (without communication between the 2 subscribers as described in the other answer).
For the new release of the MQTT spec (v5)* there is a new mode called "Shared Subscriptions". This allow s multiple clients to subscribe to a single topic and messages will be delivered by round robin to each client. This is more for load balancing rather than master/slave fail over.
*There are some brokers (HiveMQ, IBM MessageSight) that already support some version of Shared Subscriptions at MQTT v3.1.1, but they implement it in slightly different ways (different topic prefixes) so they are not cross compatible.
I am using rabbitmq to communicate between microservices written in ruby on rails. Each service subscribes to a topic. All services are scaled and run as multiple instances based on need.
During subscription bunny moves all the messages from the queue into unacked state. This makes other scaled instances to be just idle, since there is no message in ready state.
Is there a way to limit the number of messages a subscription can fetch, so that other instances can take the remaining messages from the queue.
Based on the information you made available, I'm assuming you're using rubybunny. If this assumption is incorrect (there are other ruby clients available for rabbitmq) let me know and/or check the documentation related to your client.
Back to rubybunny, link provided points to necessary information, quoting it:
For cases when multiple consumers share a queue, it is useful to be
able to specify how many messages each consumer can be sent at once
before sending the next acknowledgement.
In AMQP 0.9.1 parlance this is known as QoS or message prefetching.
Prefetching is configured on a per-channel basis.
To configure prefetching use the Bunny::Channel#prefetch method like so:
ch1 = connection1.create_channel
ch1.prefetch(10)
This is newbie question I'm sure, but using Mosquitto how do I subscribe to a retained message and guarantee that I only get the message once? I have a Paho based java subscriber, and in testing what I notice is, if my subscriber is brought down, and then restarted it pulls some of the same messages that it received before it came down. (Normally it would stay up forever, but stuff happens.) And it's critical the subscriber get each message only once. (For my use case there is only 1 subscriber.)
Is there something built into Mosquitto that provides this mechanism or is this something I have to build into my data schema and retrieval process?
I see there is a way of deleting a message by topic (https://lists.launchpad.net/mosquitto-users/msg00067.html) but right now my topics are pretty generic [Company]/[MAC address]/[Topic x] and it's possible to have two separate messages on the queue with the same topic, and I'm assuming if I send an empty message to a given topic I'll end up deleting all messages of said topic.
Thanks in advance for any help on this.
set the QoS level as 2.
QoS-level-2:
Exactly-once delivery.
This is the highest level that also incurs most over head interms of control messages and the need for locally storing the messages.
Exactly-once is a combination of at-least-once and at-most-once delivery guarantee.
I have created a connection pooling process in Erlang that has sub-processes ( each being a connection ). The connection pooling process (supervisor) needs to hold the state of all children sub-processes, such as a flag that indicates if the sub-process is available to be leased to a requester. This state is stored on a ETS table.
POOL-MASTER :
connection process 1
connection process 2
connection process 3
When a client requests a connection to POOL-MASTER, it must find out which connection process is available looking at ETS and fetching the state. This phase is called "get-lease". Then the state is updated. Similarly, when a client returns the connection to the pool, it uses a "return-lease" function that flags the item to be available to the next client.
I want to have the functions above "get-lease and return-lease" to be thread-safe. In other words, I want to make sure that no client is concurrently using these functions otherwise it turns out the state of the connections can be mixed up ( two clients get the same connection ). In java a synchronized method would be used for this purpose.
Is there anything in erlang that can be done to achieve this ? For instance some sort of locking mechanism on the ETS table and then relasing the lock ? Or should this be done creating a single process that handles the specific functions to be locked/unlocked and send messages to this process (assuming the messaging is single threaded ) ?
Thread-safe ? What is it ? Erlang does'nt know it :) since we work on message passing between processes. This makes sure that access to any structure (maintained by a server erlang process) will always be in serialized manner [same what Don Branson has mentioned.]
What I would have done is:
1. Create a gen server process monitored by a supervisor process.
2. This server process would be the manager of your ETS table and exposes API/methods to be called by clients for requesting and releasing connections.
3. The requests will be handled by handle_call(for sync call) or by handle_cast(for async call)
4. You might even want to implement some Timeout functionality to release connections by iterating over your ETS table and deleting from it based on some criteria
The above would work just fine giving you decent performance as well (if performance came to your mind). AND no race conditions as the accesses are serilized.
One approach would be to have a process dedicated to managing leases via messaging. Send a get_lease message to that process. It would receive the lease message, thus serializing access, and send a reply message to the requesting process when a lease becomes available. The lessee would send a return_lease message to the manager, which would add the lease back to the free-list.
The manager would also have to do something about processes that acquire a lease and fail to return it. It's a lease, so presumably there's an expiration that could be used for this, but the manager should also probably monitor the lessee and free the lease if a lessee fails.