I'm using Spring-AMQP for communication between some services. Inside the services I'm declaring consumers. One queue per consumer is used to dispatch the corresponding messages. A consumer-manager holds all known consumers. The actual message dispatching runs over a message-handler, which looks up for consumers subscribing the corresponding message-types. In almost any use-case a service subscribes a message only with one consumer.
For a special case I have to distinct not only the message-types but also the queue, from where the message is coming from.
Is is possible to retrieve somehow the queue name, from which the message was read?
Related
I have a system that relies on a message bus and broker to spread messages and tasks from producers to workers.
It benefits both from being able to do true pub/sub-type communications for the messages.
However, it also needs to communicate tasks. These should be done by a worker and reported back to the broker when/if the worker is finished with the task.
Can MQTT be used to publish this task by a producer, so that it is picked up by a single worker?
In my mind the producer would publish the task with a topic "TASK_FOR_USER_A" and there are X amount of workers subscribed to that topic.
The MQTT broker would then determine that it is a task and send it selectively to one of the workers.
Can this be done or is it outside the scope of MQTT brokers such as Mosquitto?
MQTT v5 has an optional extension called Shared Subscriptions which will deliver messages to a group of subscribers in a round robin approach. So each message will only be delivered to one of the group.
Mosquitto v1.6.x has implemented MQTT v5 and the shared subscription capability.
It's not clear what you mean by 1 message at a time. Messages will be delivered as they arrive and the broker will not wait for one subscriber to finish working on a message before delivering the next message to the next subscriber in the group.
If you have low enough control over the client then you can prevent the high QOS responses to prevent the client from acknowledging the message and force the broker to only allow 1 message to be in flight at a time which would effectively throttle message delivery, but you should only do this if message processing is very quick to prevent the broker from deciding delivery has failed and attempting to deliver the message to another client in the shared group.
Normally the broker will not do any routing above and beyond that based on the topic. The as mentioned in a comment on this answer the Flespi has implemented "sticky sessions" so that messages from a specific publisher will be delivered to the same client in the shared subscription pool, but this is a custom add on and not part of the spec.
What you're looking for is a message broker for a producer/consumer scenario. MQTT is a lightweight messaging protocol which is based on pub/sub model. If you start using any MQTT broker for this, you might face issues depending upon your use case. A few issues to list:
You need ordering of the messages (consumer must get the messages in the same order the producer published those). While QoS 2 guarantees message order without having shared subscriptions, having shared subscriptions doesn't provide ordered topic guarantees.
Consumer gets the message but fails before processing it and the MQTT broker has already acknowledged the message delivery. In this case, the consumer needs to specifically handle the reprocessing of failed messages.
If you go with a single topic with multiple subscribers, you must have idempotency in your consumer.
I would suggest to go for a message broker suitable for this purpose, e.g. Kafka, RabbitMQ to name a few.
As far as I know, MQTT is not meant for this purpose. It doesn't have any internal working to distribute the tasks on workers (consumers). On the Otherhand, AMQP can be used here. One hack would be to conditionalize the workers to accept only a particular type of tasks, but that needs producers to send task type as well. In this case, you won't be able to scale as well.
It's better if you explore other protocols for this type of usecase.
I wish to use Solace guaranteed messaging with load balanced consumers.
Currently if I use a queue with non-exclusive access type and have multiple consumers each consumer will get a message round-robin and if a consumer should die its unacknowledged messages will forwarded to another consumer.
This is the functionality I require however due to the varying size of message payloads I may exceed the limit so need to implement chunking.
The problem now is that I have no way of guaranteeing that all the chunked messages for a single message is handled by the same consumer so that it can rebuild the original message and once built do appropriate message acknowledge.
Is there a way I achieve this with Solace.
In Kafka I can use partition keys of original message to ensure chunked messages are routed to same consumer and using complicated commit sync logic.
It is possible to implement your own message chunking logic within your application using features available in Solace.
This can be achieved with the use of selectors in Solace. Egress selectors enable client applications to specify which messages they are interested in receiving, as determined by the messages’ header field and property values. Following this logic, if a message needs to be chunked, you can add the same identifier property to all of the chunks in a message. The customers will bind to the non-exclusive queue with different selectors so that each consumer will only receive messages with one type of identifier.
Another feature to consider in addition to selectors is transactions. You can start a transaction with the beginning chunk of the message and only commit when the last chunk in the message is received. This way, if the consumer should die, the transaction will rollback and all chunks in the message will be returned to the queue.
I am using rabbitmq to communicate between microservices written in ruby on rails. Each service subscribes to a topic. All services are scaled and run as multiple instances based on need.
During subscription bunny moves all the messages from the queue into unacked state. This makes other scaled instances to be just idle, since there is no message in ready state.
Is there a way to limit the number of messages a subscription can fetch, so that other instances can take the remaining messages from the queue.
Based on the information you made available, I'm assuming you're using rubybunny. If this assumption is incorrect (there are other ruby clients available for rabbitmq) let me know and/or check the documentation related to your client.
Back to rubybunny, link provided points to necessary information, quoting it:
For cases when multiple consumers share a queue, it is useful to be
able to specify how many messages each consumer can be sent at once
before sending the next acknowledgement.
In AMQP 0.9.1 parlance this is known as QoS or message prefetching.
Prefetching is configured on a per-channel basis.
To configure prefetching use the Bunny::Channel#prefetch method like so:
ch1 = connection1.create_channel
ch1.prefetch(10)
I have services A, B and C (and possibly more) and I may have NA, NB and NC running instances of each.
I want to be able to notify services A, B and C (and every other that may exist) that a given event has occurred (as in fanout), but I want that only exactly one instance of each service receives the notification (as in a work queue).
Can this be achieved with Spring AMQP?
Each your service must provide its own queue binded to that fanout exchange. Independently of instance (but not service) the queue must be the same. So, it will look like several consumer on the same queue, but only one of them will receive a message.
Frankly, you don't need to think about duplications at all. Just develop the app like it will be as only instance. And distribute it in cluster. Only one consumer (per service) will handle your message.
we are implementing (or more reimplementing) a distributed software system. What we have are different processes (possibly running on different computers) that should communicate with each other (let's call these clients). We don't want them to directly communicate with each other, but instead use some kind of message broker.
Since we like to avoid implementing the message broker ourselves we would like to use an existing implementation. But we don't find a protocol or system that fully fulfilles our requirements.
MQTT with its publish-subscribe-mechanism seems nice and could even be used for point-to-point communication (where some specific topics are only subscribed by certain clients).
But it is (like JSM, STOMP, etc.) asynchronous. The sender sends a message into the broker and doesn't know whether it is ever delivered to it's recipient. We want that the sender gets informed about a successful delivery or an elapsed timeout (when no one is receiving the message).
Is there some protocol/implementation available that provides such synchronous messaging functionality?
(It would be nice however if asynchronous delivery would be possible, too)
The messaging by default is ( usually ) asynchronous .
You can considerer RabbitMQ, it contains the following features:
Publisher-confirms (in asynchronous way):
http://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/
Transaction Commit:
https://www.rabbitmq.com/semantics.html
Messages TTL (to handle time out)
https://www.rabbitmq.com/ttl.html
With this features you can handle the time-out situations and the successful delivery.
If this is not enough you can use the RPC:
https://www.rabbitmq.com/tutorials/tutorial-six-java.html
Let me know if you need more information.