I have following usage pattern in an application:
The publisher sends messages to a topic with rate 5 microseconds per message (i.e. send one message every 5 micros).
Consumer subscribes to the topic and handles messages with rate 10 microseconds per message (i.e. it takes 10 micros to complete onReceive callback in JAVA API).
The consumer is interested only in the last message published to the topic, so all intermediate not handled messages can be dropped.
Is it possible to avoid the congestion in the queue of unprocessed messages on the consumer side?
I tried to use eliding with delay=0 (documentation link), however, it doesn’t help if the message already put to the internal queue in the Solace on the consumer side (I refer to com.solacesystems.jcsmp.impl.XMLMessageQueue).
Setting delay to some specific value works fine, but it doesn't scale well, because this number is dynamic and depends on the number of publishers and consumer performance.
A possible solution to this would be to create a LVQ (last value queue) which subscribes to the topic. You create a LVQ by setting the queue quota to 0 MB.
Then have your subscribing application consume messages from the LVQ.
In the appliance you should see the nearly same performance as when sending using direct messages as it will never hit the spool.
Related
I would like read messages from a MQTT broker ordered (chronologically as written) from more than one topic. For example I have topics (which are published to independently by clients at different rates all with QoS 2):
/foo/a
/foo/b
/foo/c
The messages are in a Persistent Session for a long period using Message Expiry Interval and the subscriber could come and go, on and offline, with any number of messages on each topic not yet read.
When I subscribe to: /foo/#, will I receive messages from topics /foo/a interleaved with messages from /foo/b and /foo/c in the order they were received by the broker ?
The specification on Message Ordering says:
... When a Server processes a message that has been published to an Ordered Topic, it MUST send PUBLISH packets to consumers (for the same Topic and QoS) in the order that they were received from any given Client [MQTT-4.6.0-5 ...
"(for the same Topic and QoS)" suggests ordering can only be guaranteed for the same Topic and QoS. So the answer my question of ordering across topics seems to be undefined.. ?
From the Mosquitto broker point of view, if a client has disconnected but has a long session expiry time and has active subscriptions with QoS>0, that is no different to a client that is connected - the session remains open. That means the messages will be delivered according to the ordering requirements in the spec.
This part of the answer covers retained messages only:
My understanding is that message ordering rules only apply for active sessions. That is to say, a client publishes messages and they must be delivered to current consumers only in the same order they were received.
It does not, however, apply to the situation when a client subscribes to a topic filter and receives retained messages. You can get a clue to the intent of the spec there, because the concept of messages being out of order for the same topic and QoS is nonsensical when there is only a single retained message per topic.
Ordering of delivery of retained messages that match a wildcard subscription is undefined. In Mosquitto is it roughly in order of delivery, breadth then depth. This is likely to change in the future to being sorted though.
I have a system that relies on a message bus and broker to spread messages and tasks from producers to workers.
It benefits both from being able to do true pub/sub-type communications for the messages.
However, it also needs to communicate tasks. These should be done by a worker and reported back to the broker when/if the worker is finished with the task.
Can MQTT be used to publish this task by a producer, so that it is picked up by a single worker?
In my mind the producer would publish the task with a topic "TASK_FOR_USER_A" and there are X amount of workers subscribed to that topic.
The MQTT broker would then determine that it is a task and send it selectively to one of the workers.
Can this be done or is it outside the scope of MQTT brokers such as Mosquitto?
MQTT v5 has an optional extension called Shared Subscriptions which will deliver messages to a group of subscribers in a round robin approach. So each message will only be delivered to one of the group.
Mosquitto v1.6.x has implemented MQTT v5 and the shared subscription capability.
It's not clear what you mean by 1 message at a time. Messages will be delivered as they arrive and the broker will not wait for one subscriber to finish working on a message before delivering the next message to the next subscriber in the group.
If you have low enough control over the client then you can prevent the high QOS responses to prevent the client from acknowledging the message and force the broker to only allow 1 message to be in flight at a time which would effectively throttle message delivery, but you should only do this if message processing is very quick to prevent the broker from deciding delivery has failed and attempting to deliver the message to another client in the shared group.
Normally the broker will not do any routing above and beyond that based on the topic. The as mentioned in a comment on this answer the Flespi has implemented "sticky sessions" so that messages from a specific publisher will be delivered to the same client in the shared subscription pool, but this is a custom add on and not part of the spec.
What you're looking for is a message broker for a producer/consumer scenario. MQTT is a lightweight messaging protocol which is based on pub/sub model. If you start using any MQTT broker for this, you might face issues depending upon your use case. A few issues to list:
You need ordering of the messages (consumer must get the messages in the same order the producer published those). While QoS 2 guarantees message order without having shared subscriptions, having shared subscriptions doesn't provide ordered topic guarantees.
Consumer gets the message but fails before processing it and the MQTT broker has already acknowledged the message delivery. In this case, the consumer needs to specifically handle the reprocessing of failed messages.
If you go with a single topic with multiple subscribers, you must have idempotency in your consumer.
I would suggest to go for a message broker suitable for this purpose, e.g. Kafka, RabbitMQ to name a few.
As far as I know, MQTT is not meant for this purpose. It doesn't have any internal working to distribute the tasks on workers (consumers). On the Otherhand, AMQP can be used here. One hack would be to conditionalize the workers to accept only a particular type of tasks, but that needs producers to send task type as well. In this case, you won't be able to scale as well.
It's better if you explore other protocols for this type of usecase.
I wish to use Solace guaranteed messaging with load balanced consumers.
Currently if I use a queue with non-exclusive access type and have multiple consumers each consumer will get a message round-robin and if a consumer should die its unacknowledged messages will forwarded to another consumer.
This is the functionality I require however due to the varying size of message payloads I may exceed the limit so need to implement chunking.
The problem now is that I have no way of guaranteeing that all the chunked messages for a single message is handled by the same consumer so that it can rebuild the original message and once built do appropriate message acknowledge.
Is there a way I achieve this with Solace.
In Kafka I can use partition keys of original message to ensure chunked messages are routed to same consumer and using complicated commit sync logic.
It is possible to implement your own message chunking logic within your application using features available in Solace.
This can be achieved with the use of selectors in Solace. Egress selectors enable client applications to specify which messages they are interested in receiving, as determined by the messages’ header field and property values. Following this logic, if a message needs to be chunked, you can add the same identifier property to all of the chunks in a message. The customers will bind to the non-exclusive queue with different selectors so that each consumer will only receive messages with one type of identifier.
Another feature to consider in addition to selectors is transactions. You can start a transaction with the beginning chunk of the message and only commit when the last chunk in the message is received. This way, if the consumer should die, the transaction will rollback and all chunks in the message will be returned to the queue.
I am using rabbitmq to communicate between microservices written in ruby on rails. Each service subscribes to a topic. All services are scaled and run as multiple instances based on need.
During subscription bunny moves all the messages from the queue into unacked state. This makes other scaled instances to be just idle, since there is no message in ready state.
Is there a way to limit the number of messages a subscription can fetch, so that other instances can take the remaining messages from the queue.
Based on the information you made available, I'm assuming you're using rubybunny. If this assumption is incorrect (there are other ruby clients available for rabbitmq) let me know and/or check the documentation related to your client.
Back to rubybunny, link provided points to necessary information, quoting it:
For cases when multiple consumers share a queue, it is useful to be
able to specify how many messages each consumer can be sent at once
before sending the next acknowledgement.
In AMQP 0.9.1 parlance this is known as QoS or message prefetching.
Prefetching is configured on a per-channel basis.
To configure prefetching use the Bunny::Channel#prefetch method like so:
ch1 = connection1.create_channel
ch1.prefetch(10)
If I use CometD long polling:
Suppose there are 1000 messages in a second to be sent to subscribers, does CometD allow them to be auto-batched so that each client doesn't have to re-connect for each single message?
Do "lazy channels" (as described here: http://docs.cometd.org/3/reference/#_java_server_lazy_messages) auto-batch queued messages sent to clients upon timeout?
If on the other hand I don't use lazy channels, and suppose I "batch-publish" messages on channels 1, 2 and 3:
cometd.batch(function()
{
cometd.publish('/channel1', { product: 'foo' });
cometd.publish('/channel2', { notificationType: 'all' });
cometd.publish('/channel3', { update: false });
});
(http://docs.cometd.org/3/reference/#_javascript_batch)
does a client subscribed to all 3 channels receive them in a batch too? Or does it send them all separately, forcing the client to re-connect after each message (slow)?
CometD offers application developers full control of batching features, allowing to have maximum flexibility, performance and scalability.
When using the HTTP long-polling transports, there are 2 places where batching may happen.
From client to server is solved using the CometD API and explicit batching (like your snippet above).
Batching at this level is typically in control of the application, although CometD does an internal batching to avoid exhausting the connections to the server.
From server to client there are more variations.
For broadcast non-lazy channels there is no automation, and what normally happens is that the first message to a client (that is not the publisher) will trigger the flush of the message queue; while this is being sent, other messages will queue up on the server side for that client and on the next /meta/connect the whole queue will be flushed. For 10 messages the scheme could be something like: 1-flush-9-flush (enqueue 1, flush the queue, enqueue the other 9 while waiting for the /meta/connect to come back, flush the other 9).
For broadcast lazy channels there is automation, so CometD will wait before sending those messages following the rules of lazy messages. A typical scheme could be: 10-flush.
For service channels, everything is back in control of the application.
The client can send batched messages to the application via a service channel (whose messages are not broadcast automatically by CometD). The application on server can receive the first message and know that other 9 will come, so it can wait to send them until the last has arrived. When the last arrives, it can just use the batching API to batch together the responses to clients, something like:
List<ServerSession> subscribers = ...;
for (ServerSession subscriber : subscribers) {
subscriber.batch(() -> {
subscriber.deliver(sender, "/response", response1);
subscriber.deliver(sender, "/response", response2);
subscriber.deliver(sender, "/response", response3);
});
}
Of course responses may be different from the messages received, both in content and number.
The scheme here can be almost anything the application wants, but it's common to have it as a 10-flush, which is the most efficient.
A note regarding the batching of messages to be sent back to the publisher. This is a special case and it's by default automated: while processing the incoming messages from that publisher, CometD starts an internal batch for that particular publisher, so that any message that is to be delivered back to the publisher is batched and will be flushed at the end of the processing of the incoming messages.
The bottom line is that CometD is already pretty well tuned to give the maximum of performance and scalability in common cases, but yet leaves the application room for customizing the behaviour to achieve maximum efficiency using application specific knowledge of message patterns.
I encourage you to look at the CometD documentation, tutorials and javadocs.