I am using Spring Boot 2.1.3 with the dependency
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
I configured a single Rabbit consumer in my project, and tried setting the concurrency both through application.properties:
spring.rabbitmq.listener.simple.concurrency=4
and through the concurrency attribute of the #RabbitListener annotation.
In both cases I see exactly the same behaviour (I am running the Spring Boot application on Windows): if the queue being listened to already contains messages, but is not currently receiving any, only one thread begins processing the messages from the queue.
If then I send some new messages to the queue, indeed additional threads are started to process the messages (so that their total number is 4 in my case as specified).
Is there some additional configuration to do in order to start processing the messages already in a queue which is not currently receiving messages, by the specified number of threads?
Related
I have a Grails application using the grails RabbitMQ plugin to handle messages asynchronously. The queue is set to durable, the messages are persistent, and there are 20 concurrent consumers. Acknowledgement is turned on and is set to issue an ack/nack based on if the consumer returns normally or throws an exception. These consumers usually handle messages fine, but when the queue fills up very quickly (5,000 or so messages at once) some of the messages get lost.
There is logging in the consumer when the message from Rabbit is received and that logging event never occurs, so the consumer is not receiving the lost messages at all. Further, there are no exceptions that appear in the logs.
I have tried increasing the prefetch value of the consumers to 5 (from 1), but that did not solve the problem. I have checked the RabbitMQ UI and there are no messages stuck in the queue and there are no unacknowledged messages.
We are using spring AMQP to listen to rabbitMQ for messages. I want to be able to report the metrics once we finished processing batch of messages, that means when we exhausts all the message in the queue. I m not sure how to do that in Spring AMQP. browsing spring document, it mentions advice chain to SimpleRabbitListenerContainerFactory, but that's mainly for RetryInterceptor. is there anyway allow me to report?
There is nothing in the framework to notify the listener that there are no new messages available.
You could examine the queue using rabbitadmin to see a message count but that would be expensive to do it on every message delivery.
Some ideas:
You could schedule a task to run after some period when no messages are received (and cancel/reschedule each time a new message arrives).
You could have the sending system add a marker to the "last" message so the receiver knows the batch is complete.
Instead of using the message listener container, use RabbitTemplate.receive() (or receiveAndConvert()) which, by default, returns null when there are no messages in the queue. Call them in a loop until there are no messages. When that happens, issue your report, then go into a polling loop (with a sleep) to poll for the next "batch".
I would like to know the advantages and disadvantages of using SimpleMessageListenerContainer over receiving a message manually using Spring AMQP. Another question is when we create SimpleMessageListenerContainer setting a queue, does the rabbitmq calls the listeneradaptor or does SimpleMessageListenerContainer keeps polling the queue to check for messages and calls the registered adaptor when their is message.
It depends on your requirements; the listener container gives you an async (message-driven) approach. Otherwise, if you use the RabbitTemplatereceive methods, you are polling the queue. The container does not poll the queue, the broker pushes messages to the container according to the prefetch settings (default 1) - if using ackmode AUTO.
I am using BPEL process (AIA) to pick message from MQ (Message Queue). MQ Adapter picks message from the queue against a defined schema(nxsd). The nxsd schema has style as terminated. The issue is when an incorrect message is placed onto the queue the process continuously picks message from the queue rejects it and places it back onto the queue. This happens continuously.
Please help as this has caused a lot of server issues
I am not a MQ BPEL adapter guy. I come from MQ background.
MQ Adapter may have ability to route messages that are not understood by it to another queue known as a BACKOUT QUEUE (or BOQNAME in WebSphere MQ). Messages that are not understood by adapter (or for that matter JMS layer) are called poison messages.
There will also be another parameter typically called as a 'Threshold' (BOTHRESH in WebSphere MQ). This attribute tells the adapter when to route the message to a backout queue. For example if the attribute is set to 3, then if the same message is received 3 times (and is sent back to the same queue because the adapter does not understands this message), then when the message arrives for the fourth time, the message is automatically moved to backout queue.
My suggestion would be to look into MQ BPEL adapter documentation to see if it supports Backout queue and threshold attributes. If yes, set these attributes appropriately and try.
i have a Spring JMS based client that asynchronously listens for "triggers" on QUEUE1, they indicate that there is a message ready to be consumed on another queue, QUEUE2.
Consumption on QUEUE2 is done with JmsTemplate class, configured as follows:
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<constructor-arg ref="gpyConnectionFactory" />
<property name="destinationResolver" ref="jndiDestinationResolver" />
<property name="receiveTimeout" value="100" />
</bean>
Notice the little receiveTimeout. This value was already so, before taking charge of this application.
NOW, I noticed that sometimes, specifically when QUEUE2 contains a relative big message,
a call to
jmsTemplate.receiveSelectedAndConvert(destinationName, mqSelector);
retrieves a NULL object, so it is likely that timeout expires!
As far as I know, as JMS spec states (correct me if I'm wrong) timeout would expire only if no message is available on the queue.
The current scenario makes me believe that due to message size and due to the fact that for sure there is a message of that queue, the timeout expires because the consumer doesn't have enough time to read the whole big message.
Is it all that possible?
The provider is WebSphere MQ.
For sure I will set an higher timeout value.
The timeout is not processed by Spring, it is handled in the vendor JMS library...return consumer.receive(timeout).
The broker "pushes" the message to the consumer when it arrives in the queue but, yes, it will take a finite time to transfer a large message and it is certainly possible for the consumer.receive() operation to timeout (maybe repeatedly) until the message is fully transferred.
It's up to the vendor code to actually do the processing, but I would doubt any would block a receive because a message is in the process of being transferred.
So, putting a message in one queue is not a reliable way to notify that a message is available in another queue.
Consider just receiving from the real queue (or use a message-driven approach instead of the JmsTemplate).