Is there a limit to the number of producers and consumers on a single AWS SQS queue?
I'm being told that a queue will only support a maximum of 50 connections from producers and consumers total. That being said I can not find any documentation that supports that claim.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-throughput-horizontal-scaling-and-batching.html
Note
As you scale horizontally, you must ensure that your Amazon SQS client has enough connections or threads to support the number of concurrent message producers and consumers that send requests and receive responses. For example, by default, instances of the AWS SDK for Java AmazonSQSClient class maintain at most 50 connections to Amazon SQS. To create additional concurrent producers and consumers, you must adjust the maximum number of allowable producer and consumer threads on an AmazonSQSClientBuilder object, for example:
final AmazonSQS sqsClient = AmazonSQSClientBuilder.standard()
.withClientConfiguration(new ClientConfiguration()
.withMaxConnections(producerCount + consumerCount))
.build();
For AmazonSQSAsyncClient, you also must make sure that enough threads are available.
Related
I'm running mosquttio as an MQTT broker and I have multiple devices sending sensory data periodically. I want to collect all the messages and store them. My question is, is there any advantage to having multiple connections (each connection has a unique id and subscribed to subset of the topics.) to the broker or is it preferable to have a single connection gathering all the data.
Note: the subscribers will be on the same machine as the broker.
It probably depends on the messages in question and what processing the client is going to do on those messages.
Having a single client subscribed to '#' will mean that there will only be a single entry on the subscribed topic to search for matches when processing a message. But this is probably a negligible amount of overhead under most situations.
If the message rate is high enough and there is any overhead to the storage then using something called Shared Subscriptions to allow a pool of client to all subscribe to the same topic (or wildcard) and ensure that any message is only delivered to a single client in the pool. This means that the processing of messages can be load balanced over the pool of clients.
Using Shared Subscriptions means that you can dynamically add or remove clients from the pool without having to repartition the topic space across the clients.
I have an issue where I have backend server and thousands of clients.
Each client has its own topic in MQTT.
Communication is bi-directional: clients can ask backend for smth, backend can respond or notify by some action in real-time.
How should I scale my backend subscribers to process a huge amount of messages from MQTT?
As MQTT implements pub/sub pattern, when I will scale subscribers to process more messages at the same time by adding one more instance, it will subscribe to the same topic and receive the same message as other subscribers.
Pub/sub scaling issue: Subscriber1, Subscriber2 will get Message1, then Subscriber1, Subscriber2 will get Message2.
It is opposite to AMQP when I have a consumer of the queue instead of pub/sub.
Consumer1 will get Message1, Consumer2 will get Message2, so scaling is efficient.
So is it a good choice for backend server to use MQTT in case of real-time communication with a huge amount of clients? How to deal with it?
I believe that shared subscriptions, a feature of MQTT v5, address your concern:
Like a Non‑shared Subscription, it has a Topic Filter and Subscription Options; however, a publication that matches its Topic Filter is only sent to one of its subscribing Sessions. Shared Subscriptions are useful where several consuming Clients share the processing of the publications in parallel.
I have a system that relies on a message bus and broker to spread messages and tasks from producers to workers.
It benefits both from being able to do true pub/sub-type communications for the messages.
However, it also needs to communicate tasks. These should be done by a worker and reported back to the broker when/if the worker is finished with the task.
Can MQTT be used to publish this task by a producer, so that it is picked up by a single worker?
In my mind the producer would publish the task with a topic "TASK_FOR_USER_A" and there are X amount of workers subscribed to that topic.
The MQTT broker would then determine that it is a task and send it selectively to one of the workers.
Can this be done or is it outside the scope of MQTT brokers such as Mosquitto?
MQTT v5 has an optional extension called Shared Subscriptions which will deliver messages to a group of subscribers in a round robin approach. So each message will only be delivered to one of the group.
Mosquitto v1.6.x has implemented MQTT v5 and the shared subscription capability.
It's not clear what you mean by 1 message at a time. Messages will be delivered as they arrive and the broker will not wait for one subscriber to finish working on a message before delivering the next message to the next subscriber in the group.
If you have low enough control over the client then you can prevent the high QOS responses to prevent the client from acknowledging the message and force the broker to only allow 1 message to be in flight at a time which would effectively throttle message delivery, but you should only do this if message processing is very quick to prevent the broker from deciding delivery has failed and attempting to deliver the message to another client in the shared group.
Normally the broker will not do any routing above and beyond that based on the topic. The as mentioned in a comment on this answer the Flespi has implemented "sticky sessions" so that messages from a specific publisher will be delivered to the same client in the shared subscription pool, but this is a custom add on and not part of the spec.
What you're looking for is a message broker for a producer/consumer scenario. MQTT is a lightweight messaging protocol which is based on pub/sub model. If you start using any MQTT broker for this, you might face issues depending upon your use case. A few issues to list:
You need ordering of the messages (consumer must get the messages in the same order the producer published those). While QoS 2 guarantees message order without having shared subscriptions, having shared subscriptions doesn't provide ordered topic guarantees.
Consumer gets the message but fails before processing it and the MQTT broker has already acknowledged the message delivery. In this case, the consumer needs to specifically handle the reprocessing of failed messages.
If you go with a single topic with multiple subscribers, you must have idempotency in your consumer.
I would suggest to go for a message broker suitable for this purpose, e.g. Kafka, RabbitMQ to name a few.
As far as I know, MQTT is not meant for this purpose. It doesn't have any internal working to distribute the tasks on workers (consumers). On the Otherhand, AMQP can be used here. One hack would be to conditionalize the workers to accept only a particular type of tasks, but that needs producers to send task type as well. In this case, you won't be able to scale as well.
It's better if you explore other protocols for this type of usecase.
I have following usage pattern in an application:
The publisher sends messages to a topic with rate 5 microseconds per message (i.e. send one message every 5 micros).
Consumer subscribes to the topic and handles messages with rate 10 microseconds per message (i.e. it takes 10 micros to complete onReceive callback in JAVA API).
The consumer is interested only in the last message published to the topic, so all intermediate not handled messages can be dropped.
Is it possible to avoid the congestion in the queue of unprocessed messages on the consumer side?
I tried to use eliding with delay=0 (documentation link), however, it doesn’t help if the message already put to the internal queue in the Solace on the consumer side (I refer to com.solacesystems.jcsmp.impl.XMLMessageQueue).
Setting delay to some specific value works fine, but it doesn't scale well, because this number is dynamic and depends on the number of publishers and consumer performance.
A possible solution to this would be to create a LVQ (last value queue) which subscribes to the topic. You create a LVQ by setting the queue quota to 0 MB.
Then have your subscribing application consume messages from the LVQ.
In the appliance you should see the nearly same performance as when sending using direct messages as it will never hit the spool.
I am using rabbitmq to communicate between microservices written in ruby on rails. Each service subscribes to a topic. All services are scaled and run as multiple instances based on need.
During subscription bunny moves all the messages from the queue into unacked state. This makes other scaled instances to be just idle, since there is no message in ready state.
Is there a way to limit the number of messages a subscription can fetch, so that other instances can take the remaining messages from the queue.
Based on the information you made available, I'm assuming you're using rubybunny. If this assumption is incorrect (there are other ruby clients available for rabbitmq) let me know and/or check the documentation related to your client.
Back to rubybunny, link provided points to necessary information, quoting it:
For cases when multiple consumers share a queue, it is useful to be
able to specify how many messages each consumer can be sent at once
before sending the next acknowledgement.
In AMQP 0.9.1 parlance this is known as QoS or message prefetching.
Prefetching is configured on a per-channel basis.
To configure prefetching use the Bunny::Channel#prefetch method like so:
ch1 = connection1.create_channel
ch1.prefetch(10)